[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:46 dist-test-slave-c0l9 krb5kdc[6190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:46 dist-test-slave-c0l9 krb5kdc[6190](info): set up 2 sockets
Sep 29 11:32:46 dist-test-slave-c0l9 krb5kdc[6190](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 11:32:47.572572 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:32:47 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:32:47.732975 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43375
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:48.073817 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:48.074152 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:48.074283 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:48.081705 6203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:48.081828 6203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:48.081907 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:48.081987 6203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:48.082056 6203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:48.092700 6203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:48.094285 6203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:48.095830 6203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:48.109143 6209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:48.109110 6211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:48.109355 6203 server_base.cc:1034] running on GCE node
W20260929 11:32:48.109164 6208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:48.110677 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:48.112294 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:48.113512 6203 hybrid_clock.cc:648] HybridClock initialized: now 1790681568113453 us; error 46 us; skew 500 ppm
I20260929 11:32:48.113899 6203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:48 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:48.117158 6203 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:32:48.118973 6203 webserver.cc:460] Webserver started at http://127.6.9.254:44011/ using document root <none> and password file <none>
I20260929 11:32:48.119740 6203 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:48.119868 6203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:48.120198 6203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:48.123570 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "c647741117354e80baee1928c44e6e71"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "ba7abdc73177a65b3901404741190fcb"
server_key_iv: "d7a257e346c3f138c1e3becc5dfe6128"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.124396 6203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "c647741117354e80baee1928c44e6e71"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "ba7abdc73177a65b3901404741190fcb"
server_key_iv: "d7a257e346c3f138c1e3becc5dfe6128"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.130681 6203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 11:32:48.134958 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:48.136751 6203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:48.136982 6203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c647741117354e80baee1928c44e6e71"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "ba7abdc73177a65b3901404741190fcb"
server_key_iv: "d7a257e346c3f138c1e3becc5dfe6128"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.137192 6203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:48.156701 6203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:48.161350 6203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:48.161640 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:48.180122 6203 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43375
I20260929 11:32:48.180181 6269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43375 every 8 connection(s)
I20260929 11:32:48.181598 6203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:48.185779 6270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:48.189623 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6203
I20260929 11:32:48.189917 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:32:48.192005 6183 external_mini_cluster.cc:1428] Setting key 905097ed1b5d8c71132b6a6d6b3325e1
I20260929 11:32:48.197984 6270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71: Bootstrap starting.
I20260929 11:32:48.202479 6270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:48.204034 6270 log.cc:824] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:48.207952 6270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71: No bootstrap required, opened a new log
I20260929 11:32:48.219485 6270 raft_consensus.cc:348] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } }
I20260929 11:32:48.219914 6270 raft_consensus.cc:374] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:48.220073 6270 raft_consensus.cc:729] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c647741117354e80baee1928c44e6e71, State: Initialized, Role: FOLLOWER
I20260929 11:32:48.220861 6270 consensus_queue.cc:260] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } }
I20260929 11:32:48.221154 6270 raft_consensus.cc:388] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:48.221310 6270 raft_consensus.cc:482] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:48.221499 6270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:48.224421 6270 raft_consensus.cc:504] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } }
I20260929 11:32:48.225234 6270 leader_election.cc:302] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c647741117354e80baee1928c44e6e71; no voters:
I20260929 11:32:48.226109 6270 leader_election.cc:288] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:48.226402 6275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:48.227684 6275 raft_consensus.cc:686] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [term 1 LEADER]: Becoming Leader. State: Replica: c647741117354e80baee1928c44e6e71, State: Running, Role: LEADER
I20260929 11:32:48.228334 6275 consensus_queue.cc:237] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } }
I20260929 11:32:48.228721 6270 sys_catalog.cc:564] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 11:32:48 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:48.231703 6277 sys_catalog.cc:455] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c647741117354e80baee1928c44e6e71. Latest consensus state: current_term: 1 leader_uuid: "c647741117354e80baee1928c44e6e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } } }
I20260929 11:32:48.232188 6277 sys_catalog.cc:458] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:48.232770 6276 sys_catalog.cc:455] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c647741117354e80baee1928c44e6e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c647741117354e80baee1928c44e6e71" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43375 } } }
I20260929 11:32:48.233119 6276 sys_catalog.cc:458] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:48.240299 6281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:48.244874 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:48.196782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.6.9.254:43375)
0929 11:32:48.197748 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:48.197766 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:48.199896 (+ 2130us) server_negotiation.cc:407] Connection header received
0929 11:32:48.213576 (+ 13680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:48.213608 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:48.214133 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:48.214574 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:48.219249 (+ 4675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.220815 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.223523 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.224291 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.224494 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:48.232983 (+ 8489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:48.233008 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:48.233031 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:48.233101 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:48.237664 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:48.239024 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:48.239035 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:48.239046 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:48.239335 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:48.240089 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:48.240098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:48.240101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:48.241139 (+ 1038us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:48.241482 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:48.242663 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 11:32:48.243147 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":401,"threads_started":1}
I20260929 11:32:48.248284 6281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:48.258735 6281 catalog_manager.cc:1347] Generated new cluster ID: 3e9df4dccc0e44b889939c53fe179919
I20260929 11:32:48.258888 6281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:48.271515 6281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:48.273312 6281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:48.285583 6281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c647741117354e80baee1928c44e6e71: Generated new TSK 0
I20260929 11:32:48.286528 6281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:32:48.331575 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:48.719797 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:48.720116 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:48.720228 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:48.727609 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:48.727730 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:48.727844 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:32:48.737818 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:48.739101 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:48.740443 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:48.753965 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:48.754052 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:48.754004 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:48.754799 6297 server_base.cc:1034] running on GCE node
I20260929 11:32:48.755461 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:48.756450 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:48.757654 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790681568757582 us; error 60 us; skew 500 ppm
I20260929 11:32:48.757980 6297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:48 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:48.761149 6297 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:32:48.762563 6297 webserver.cc:460] Webserver started at http://127.6.9.193:37295/ using document root <none> and password file <none>
I20260929 11:32:48.763233 6297 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:48.763350 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:48.763649 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:48.766815 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "370049d0f0c14b72b61b5113093adb38"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "5bfe2f6649268a7fc73c1499ced66a01"
server_key_iv: "b11a6010d527f9e3c361c152a89a74b4"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.767709 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "370049d0f0c14b72b61b5113093adb38"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "5bfe2f6649268a7fc73c1499ced66a01"
server_key_iv: "b11a6010d527f9e3c361c152a89a74b4"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.773465 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:32:48.777093 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:48.778584 6297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:32:48.778785 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "370049d0f0c14b72b61b5113093adb38"
format_stamp: "Formatted at 2026-09-29 11:32:48 on dist-test-slave-c0l9"
server_key: "5bfe2f6649268a7fc73c1499ced66a01"
server_key_iv: "b11a6010d527f9e3c361c152a89a74b4"
server_key_version: "encryptionkey@0"
I20260929 11:32:48.778995 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:48.796583 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:48.801321 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:48.801609 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:48.802742 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:48.804210 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:48.804288 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:48.804425 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:48.804477 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:48.835080 6297 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:36907
I20260929 11:32:48.835122 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:36907 every 8 connection(s)
I20260929 11:32:48.836351 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:32:48.841774 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6297
I20260929 11:32:48.841975 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:32:48.842613 6183 external_mini_cluster.cc:1428] Setting key 71d4054c630ca055ed163eb3e4fc402b
Sep 29 11:32:48 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681568, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:32:48.856426 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:48.838888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:50605 (local address 127.6.9.254:43375)
0929 11:32:48.839317 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:48.839322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:48.840199 (+ 877us) server_negotiation.cc:407] Connection header received
0929 11:32:48.842222 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:48.842228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:48.842337 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:48.842478 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:48.844111 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.845017 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.846567 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.846968 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.847083 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:48.850706 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:48.850712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:48.850715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:48.850748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:48.853335 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:48.854071 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:48.854077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:48.854080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:48.854218 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:48.854630 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:48.854633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:48.854635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:48.854913 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:48.855042 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:48.855793 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 11:32:48.856036 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260929 11:32:48.857586 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:48.839298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43375 (local address 127.6.9.193:50605)
0929 11:32:48.840024 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:48.840068 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:48.841926 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:48.842761 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:48.842772 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:48.843348 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:48.843943 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:48.843961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.845172 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.845176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:48.846427 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:48.846439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:48.847203 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:48.847210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:48.847442 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:48.848080 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:48.848108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:48.850467 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:48.853480 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:48.853490 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:48.853510 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:48.853954 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:48.854326 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:48.854332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:48.854336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:48.854527 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:48.855041 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:48.855051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:48.855413 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 11:32:48.855684 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 11:32:48.856269 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":284,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
I20260929 11:32:48.859202 6425 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43375
I20260929 11:32:48.859508 6425 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:48.860278 6425 heartbeater.cc:511] Master 127.6.9.254:43375 requested a full tablet report, sending...
I20260929 11:32:48.862890 6235 ts_manager.cc:194] Registered new tserver with Master: 370049d0f0c14b72b61b5113093adb38 (127.6.9.193:36907)
I20260929 11:32:48.864426 6235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:50605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:32:48.880466 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:49.222151 6431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:49.222476 6431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:49.222636 6431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:49.230837 6431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:49.230988 6431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:49.231148 6431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:32:49.241799 6431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:49.243196 6431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:49.244529 6431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:49.257782 6439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:49.257813 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:49.257725 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:49.258805 6431 server_base.cc:1034] running on GCE node
I20260929 11:32:49.259524 6431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:49.260519 6431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:49.261739 6431 hybrid_clock.cc:648] HybridClock initialized: now 1790681569261683 us; error 61 us; skew 500 ppm
I20260929 11:32:49.262089 6431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:49.265410 6431 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:32:49.266901 6431 webserver.cc:460] Webserver started at http://127.6.9.194:33727/ using document root <none> and password file <none>
I20260929 11:32:49.267632 6431 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:49.267751 6431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:49.268069 6431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:49.271147 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "880c708013ca44d4b8d1cd82d05320eb"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "c5c785156205497cf44ac1c467d231e9"
server_key_iv: "52c447a120f74adfcfef9257a6597579"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.271938 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "880c708013ca44d4b8d1cd82d05320eb"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "c5c785156205497cf44ac1c467d231e9"
server_key_iv: "52c447a120f74adfcfef9257a6597579"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.277251 6431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:32:49.280756 6446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:49.282097 6431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:32:49.282253 6431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "880c708013ca44d4b8d1cd82d05320eb"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "c5c785156205497cf44ac1c467d231e9"
server_key_iv: "52c447a120f74adfcfef9257a6597579"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.282447 6431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:49.321202 6431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:49.325711 6431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:49.325992 6431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:49.327006 6431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:49.328372 6431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:49.328446 6431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:49.328580 6431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:49.328630 6431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:49.359210 6431 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:46309
I20260929 11:32:49.359287 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:46309 every 8 connection(s)
I20260929 11:32:49.360504 6431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:32:49.367305 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6431
I20260929 11:32:49.367476 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:32:49.367916 6183 external_mini_cluster.cc:1428] Setting key efedaf3f482f6356de60ebee4df81bc3
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:49.380986 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.363129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34983 (local address 127.6.9.254:43375)
0929 11:32:49.363600 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:49.363606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:49.364586 (+ 980us) server_negotiation.cc:407] Connection header received
0929 11:32:49.366417 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:49.366421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:49.366502 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:49.366601 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:49.368190 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.369084 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.370690 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.371104 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.371212 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.374627 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:49.374634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:49.374639 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:49.374679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:49.377817 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.378571 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.378576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.378580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.378755 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.379201 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.379204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.379207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.379486 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:49.379618 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:49.380323 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 11:32:49.380562 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260929 11:32:49.381999 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.363683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43375 (local address 127.6.9.194:34983)
0929 11:32:49.364415 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:49.364460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:49.366148 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:49.366800 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:49.366808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:49.367352 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:49.368002 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:49.368026 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.369231 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.369235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:49.370542 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:49.370566 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.371325 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.371334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:49.371562 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.372195 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:49.372218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:49.374401 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:49.377981 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:49.377988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:49.378003 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:49.378447 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:49.378866 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:49.378869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:49.378872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:49.379082 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:49.379618 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:49.379628 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:49.379961 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 11:32:49.380208 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 11:32:49.380806 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:32:49.383349 6559 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43375
I20260929 11:32:49.383677 6559 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:49.384460 6559 heartbeater.cc:511] Master 127.6.9.254:43375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:32:49.386085 6235 ts_manager.cc:194] Registered new tserver with Master: 880c708013ca44d4b8d1cd82d05320eb (127.6.9.194:46309)
I20260929 11:32:49.386942 6235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:34983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:32:49.408228 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:49.729413 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:49.729740 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:49.729887 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:49.737373 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:49.737517 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:49.737672 6565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:32:49.747848 6565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:49.749262 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:49.750597 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:49.764319 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:49.764379 6573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:49.764350 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:49.764863 6565 server_base.cc:1034] running on GCE node
I20260929 11:32:49.765522 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:49.766477 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:49.767712 6565 hybrid_clock.cc:648] HybridClock initialized: now 1790681569767641 us; error 54 us; skew 500 ppm
I20260929 11:32:49.768085 6565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:49.771389 6565 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:32:49.773046 6565 webserver.cc:460] Webserver started at http://127.6.9.195:43683/ using document root <none> and password file <none>
I20260929 11:32:49.773775 6565 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:49.773926 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:49.774255 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:49.777532 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "f149569b982e41f0a752f2a04d9c0f62"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "f4be5868eea613cb193ab37f622c170d"
server_key_iv: "c9b3a21a55dd6e9910dc74897b0cbf55"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.778318 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "f149569b982e41f0a752f2a04d9c0f62"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "f4be5868eea613cb193ab37f622c170d"
server_key_iv: "c9b3a21a55dd6e9910dc74897b0cbf55"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.784140 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:32:49.787998 6580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:49.789525 6565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:49.789697 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "f149569b982e41f0a752f2a04d9c0f62"
format_stamp: "Formatted at 2026-09-29 11:32:49 on dist-test-slave-c0l9"
server_key: "f4be5868eea613cb193ab37f622c170d"
server_key_iv: "c9b3a21a55dd6e9910dc74897b0cbf55"
server_key_version: "encryptionkey@0"
I20260929 11:32:49.789897 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:49.825001 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:49.829586 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:49.829921 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:49.831070 6565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:49.832648 6565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:49.832758 6565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 11:32:49.832895 6565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:49.832950 6565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:49.867882 6425 heartbeater.cc:503] Master 127.6.9.254:43375 was elected leader, sending a full tablet report...
I20260929 11:32:49.879089 6565 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:43245
I20260929 11:32:49.879150 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:43245 every 8 connection(s)
I20260929 11:32:49.880631 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:32:49.884804 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6565
I20260929 11:32:49.884997 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:32:49.885489 6183 external_mini_cluster.cc:1428] Setting key de947242c48c39e13310995548063d27
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:49.903890 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.883621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:44497 (local address 127.6.9.254:43375)
0929 11:32:49.884035 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:49.884040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:49.885292 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 11:32:49.888044 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:49.888049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:49.888131 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:49.888247 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:49.890426 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.891558 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.894071 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.894648 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.894757 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.898407 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:49.898412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:49.898416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:49.898451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:49.901155 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.901948 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.901952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.901954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.902096 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.902437 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.902439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.902441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.902691 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:49.902803 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:49.903448 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 11:32:49.903606 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":213,"threads_started":1}
I20260929 11:32:49.904886 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.884235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43375 (local address 127.6.9.195:44497)
0929 11:32:49.885112 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:49.885163 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:49.887735 (+ 2572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:49.888525 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:49.888536 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:49.889392 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:49.890236 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:49.890264 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.891726 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.891733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:49.893929 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:49.893944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.894874 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.894883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:49.895204 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.895927 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:49.895956 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:49.898164 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:49.901286 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:49.901296 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:49.901316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:49.901822 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:49.902177 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:49.902181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:49.902183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:49.902359 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:49.902801 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:49.902808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:49.903142 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 11:32:49.903364 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 11:32:49.903776 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260929 11:32:49.906102 6693 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43375
I20260929 11:32:49.906396 6693 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:49.907146 6693 heartbeater.cc:511] Master 127.6.9.254:43375 requested a full tablet report, sending...
I20260929 11:32:49.908609 6235 ts_manager.cc:194] Registered new tserver with Master: f149569b982e41f0a752f2a04d9c0f62 (127.6.9.195:43245)
I20260929 11:32:49.909467 6235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:44497
I20260929 11:32:49.913879 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:32:49.956586 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.940628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.6.9.193:36907)
0929 11:32:49.941440 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:49.941448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:49.941490 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:32:49.941552 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:49.941559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:49.941829 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:49.941982 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:49.943422 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.944462 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.945856 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.946512 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.946633 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.950656 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:49.950669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:49.950685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:49.950733 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:49.953715 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.954482 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.954490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.954493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.954640 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.955107 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.955112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.955114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.955490 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:49.955728 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:49.956078 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 11:32:49.956296 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":444,"threads_started":1}
W20260929 11:32:49.957862 6340 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46826
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:32:49 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:32:49.998805 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:49.985348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.6.9.193:36907)
0929 11:32:49.985528 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:49.985535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:49.985889 (+ 354us) server_negotiation.cc:407] Connection header received
0929 11:32:49.986072 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:49.986076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:49.986145 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:49.986229 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:49.987536 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.988208 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.989529 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:49.989955 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:49.990065 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:49.993740 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:49.993745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:49.993748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:49.993782 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:49.996329 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.997008 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.997013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.997015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.997144 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:49.997498 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:49.997502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:49.997504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:49.997770 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:49.997995 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:49.998284 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 11:32:49.998497 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:32:50.000232 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6297
I20260929 11:32:50.011723 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:50.001130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.6.9.193:36907)
0929 11:32:50.001271 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:50.001275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:50.001641 (+ 366us) server_negotiation.cc:407] Connection header received
0929 11:32:50.001758 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:50.001761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:50.001813 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:50.001886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:50.002905 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.003590 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.004764 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.005219 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.005323 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:50.007337 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:50.007341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:50.007345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:50.007372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:50.009671 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.010246 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.010250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.010252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.010372 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.010635 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.010637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.010639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.010864 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:50.011056 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:50.011251 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 11:32:50.011398 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:32:50.012446 6420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:50.096128 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6297
I20260929 11:32:50.108752 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6431
Sep 29 11:32:50 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:32:50.124257 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:50.109644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.6.9.194:46309)
0929 11:32:50.110218 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:50.110226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:50.110250 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:32:50.110311 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:50.110318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:50.110513 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:50.110642 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:50.111753 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.112833 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.114087 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.114627 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.114720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:50.118974 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:50.118986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:50.119000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:50.119041 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:50.121746 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.122378 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.122384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.122386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.122495 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.122829 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.122831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.122833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.123146 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:50.123360 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:50.123683 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:32:50.123884 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":308,"threads_started":1}
I20260929 11:32:50.125593 6554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:50.210096 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6431
I20260929 11:32:50.222515 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6565
Sep 29 11:32:50 dist-test-slave-c0l9 krb5kdc[6190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:32:50.238153 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:50.223471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.6.9.195:43245)
0929 11:32:50.224114 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:50.224121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:50.224150 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:32:50.224208 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:50.224214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:50.224412 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:50.224559 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:50.225791 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.226805 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.228046 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:50.228614 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:50.228754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:50.232730 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:50.232743 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:50.232758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:50.232798 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:50.235709 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.236317 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.236323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.236326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.236452 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:50.236759 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:50.236763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:50.236766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:50.237126 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:50.237377 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:50.237584 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 11:32:50.237802 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":347,"threads_started":1}
I20260929 11:32:50.239482 6688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:50.327131 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6565
I20260929 11:32:50.339397 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6203
I20260929 11:32:50.340533 6265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:50.441087 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6203
2026-09-29T11:32:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3876 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:50 dist-test-slave-c0l9 krb5kdc[6715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:50 dist-test-slave-c0l9 krb5kdc[6715](info): set up 2 sockets
Sep 29 11:32:50 dist-test-slave-c0l9 krb5kdc[6715](info): commencing operation
krb5kdc: starting...
W20260929 11:32:51.424499 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.008s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:32:51 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:32:51.588351 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:35307
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:35307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:51.896929 6728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:51.897234 6728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:51.897367 6728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:51.904361 6728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:51.904470 6728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:51.904549 6728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:51.904624 6728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:51.904703 6728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:51.914182 6728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:35307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:35307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:51.915433 6728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:51.916683 6728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:51.927620 6733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:51.927649 6734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:51.927567 6736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:51.928332 6728 server_base.cc:1034] running on GCE node
I20260929 11:32:51.929091 6728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:51.930399 6728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:51.931588 6728 hybrid_clock.cc:648] HybridClock initialized: now 1790681571931547 us; error 33 us; skew 500 ppm
I20260929 11:32:51.931906 6728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:51 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:51.940232 6728 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:32:51.941619 6728 webserver.cc:460] Webserver started at http://127.6.9.254:42795/ using document root <none> and password file <none>
I20260929 11:32:51.942252 6728 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:51.942364 6728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:51.942649 6728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:51.945520 6728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "a69b937cb4bc401f9179285f8e5b54e5"
format_stamp: "Formatted at 2026-09-29 11:32:51 on dist-test-slave-c0l9"
server_key: "458a7284acfe646daae627c3486f0894"
server_key_iv: "a974b599d15b5c0a342ea741e3480216"
server_key_version: "encryptionkey@0"
I20260929 11:32:51.946254 6728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "a69b937cb4bc401f9179285f8e5b54e5"
format_stamp: "Formatted at 2026-09-29 11:32:51 on dist-test-slave-c0l9"
server_key: "458a7284acfe646daae627c3486f0894"
server_key_iv: "a974b599d15b5c0a342ea741e3480216"
server_key_version: "encryptionkey@0"
I20260929 11:32:51.951174 6728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:32:51.954355 6743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:51.955528 6728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:32:51.955691 6728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "a69b937cb4bc401f9179285f8e5b54e5"
format_stamp: "Formatted at 2026-09-29 11:32:51 on dist-test-slave-c0l9"
server_key: "458a7284acfe646daae627c3486f0894"
server_key_iv: "a974b599d15b5c0a342ea741e3480216"
server_key_version: "encryptionkey@0"
I20260929 11:32:51.955860 6728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:51.968230 6728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:51.971866 6728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:51.972105 6728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:51.988466 6728 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:35307
I20260929 11:32:51.988512 6794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:35307 every 8 connection(s)
I20260929 11:32:51.989756 6728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:51.992998 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6728
I20260929 11:32:51.993207 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:32:51.993408 6795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:51.993848 6183 external_mini_cluster.cc:1428] Setting key 6fa058ae86d44e4780cc0de9624522be
I20260929 11:32:52.007766 6795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5: Bootstrap starting.
Sep 29 11:32:52 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:52.011332 6795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:52.012434 6795 log.cc:824] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:52.015456 6795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5: No bootstrap required, opened a new log
I20260929 11:32:52.016322 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:51.995544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.6.9.254:35307)
0929 11:32:51.996381 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:51.996391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:51.996430 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:32:51.997903 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:51.997924 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:51.998286 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:51.998588 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:52.000482 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.001783 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.003618 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.004355 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.004501 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:52.009322 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:52.009329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:52.009343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:52.009388 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:52.012357 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:52.013084 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:52.013091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:52.013097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:52.013265 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:52.013622 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:52.013626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:52.013628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:52.014155 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:52.014416 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:52.014906 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 11:32:52.015306 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":374,"threads_started":1}
I20260929 11:32:52.024050 6795 raft_consensus.cc:348] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } }
I20260929 11:32:52.024474 6795 raft_consensus.cc:374] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:52.024588 6795 raft_consensus.cc:729] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69b937cb4bc401f9179285f8e5b54e5, State: Initialized, Role: FOLLOWER
I20260929 11:32:52.025226 6795 consensus_queue.cc:260] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } }
I20260929 11:32:52.025498 6795 raft_consensus.cc:388] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:52.025636 6795 raft_consensus.cc:482] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:52.025799 6795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:52.028375 6795 raft_consensus.cc:504] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } }
I20260929 11:32:52.028995 6795 leader_election.cc:302] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a69b937cb4bc401f9179285f8e5b54e5; no voters:
I20260929 11:32:52.029577 6795 leader_election.cc:288] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:52.029707 6800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:52.030349 6800 raft_consensus.cc:686] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [term 1 LEADER]: Becoming Leader. State: Replica: a69b937cb4bc401f9179285f8e5b54e5, State: Running, Role: LEADER
I20260929 11:32:52.030913 6800 consensus_queue.cc:237] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } }
I20260929 11:32:52.031649 6795 sys_catalog.cc:564] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:52.033639 6801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a69b937cb4bc401f9179285f8e5b54e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } } }
I20260929 11:32:52.033666 6802 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a69b937cb4bc401f9179285f8e5b54e5. Latest consensus state: current_term: 1 leader_uuid: "a69b937cb4bc401f9179285f8e5b54e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a69b937cb4bc401f9179285f8e5b54e5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35307 } } }
I20260929 11:32:52.034082 6801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:52.034160 6802 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:52.034859 6806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:52.045549 6806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:52.054160 6806 catalog_manager.cc:1347] Generated new cluster ID: f997dd29dbe6412a97e53bb8610ee311
I20260929 11:32:52.054292 6806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:52.076910 6806 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:52.078279 6806 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:52.086819 6806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a69b937cb4bc401f9179285f8e5b54e5: Generated new TSK 0
I20260929 11:32:52.087657 6806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:32:52.136354 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:52.460896 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:52.461187 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:52.461315 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:52.468451 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:52.468552 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:52.468722 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:32:52.478221 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:52.479609 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:52.480975 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:52.493216 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:52.493216 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:52.493276 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:52.494161 6821 server_base.cc:1034] running on GCE node
I20260929 11:32:52.494724 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:52.495611 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:52.496762 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790681572496702 us; error 57 us; skew 500 ppm
I20260929 11:32:52.497068 6821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:52 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:52.499986 6821 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:32:52.501468 6821 webserver.cc:460] Webserver started at http://127.6.9.193:44461/ using document root <none> and password file <none>
I20260929 11:32:52.502148 6821 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:52.502261 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:52.502547 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:52.505460 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "25ad71d80f784ee6aaa47d23c70873ea"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "8e37ceb3f507aa1d40d0b2b6b768b5a7"
server_key_iv: "220cc1544727c06bf494a6e3491e7071"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.506225 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "25ad71d80f784ee6aaa47d23c70873ea"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "8e37ceb3f507aa1d40d0b2b6b768b5a7"
server_key_iv: "220cc1544727c06bf494a6e3491e7071"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.511260 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:32:52.514856 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:52.516460 6821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:32:52.516629 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "25ad71d80f784ee6aaa47d23c70873ea"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "8e37ceb3f507aa1d40d0b2b6b768b5a7"
server_key_iv: "220cc1544727c06bf494a6e3491e7071"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.516857 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:52.549003 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:52.553222 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:52.553539 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:52.554545 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:52.555899 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:52.555982 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:52.556111 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:52.556171 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:52.587671 6821 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:41297
I20260929 11:32:52.587710 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:41297 every 8 connection(s)
I20260929 11:32:52.589032 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:32:52.594301 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6821
I20260929 11:32:52.594496 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:32:52.595161 6183 external_mini_cluster.cc:1428] Setting key a41de499df2d80376afa989c9d429f8d
Sep 29 11:32:52 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:52.608610 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:52.591414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:40603 (local address 127.6.9.254:35307)
0929 11:32:52.591783 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:52.591788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:52.592761 (+ 973us) server_negotiation.cc:407] Connection header received
0929 11:32:52.594610 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:52.594616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:52.594695 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:52.594803 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:52.596319 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.597267 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.598826 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.599443 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.599583 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:52.603003 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:52.603008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:52.603011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:52.603043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:52.605539 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:52.606329 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:52.606336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:52.606338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:52.606468 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:52.606881 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:52.606887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:52.606889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:52.607161 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:52.607300 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:52.607976 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 11:32:52.608198 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":182,"threads_started":1}
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:32:52.609547 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:52.591957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:35307 (local address 127.6.9.193:40603)
0929 11:32:52.592593 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:52.592629 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:52.594341 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:52.595021 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:52.595028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:52.595502 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:52.596132 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:52.596157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.597455 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.597460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:52.598663 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:52.598676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:52.599647 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:52.599653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:52.599839 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:52.600442 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:52.600470 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:52.602784 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:52.605692 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:52.605702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:52.605718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:52.606202 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:52.606576 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:52.606581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:52.606586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:52.606772 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:52.607286 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:52.607294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:52.607612 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 11:32:52.607867 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 11:32:52.608356 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":281,"threads_started":1}
I20260929 11:32:52.611052 6949 heartbeater.cc:347] Connected to a master server at 127.6.9.254:35307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
I20260929 11:32:52.611353 6949 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:52.612020 6949 heartbeater.cc:511] Master 127.6.9.254:35307 requested a full tablet report, sending...
I20260929 11:32:52.613973 6760 ts_manager.cc:194] Registered new tserver with Master: 25ad71d80f784ee6aaa47d23c70873ea (127.6.9.193:41297)
I20260929 11:32:52.615801 6760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:40603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:32:52.631035 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:52.936529 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:52.936887 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:52.937034 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:52.944384 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:52.944501 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:52.944684 6955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:32:52.954392 6955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:52.955644 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:52.956960 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:52.969516 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:52.969583 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:52.969601 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:52.970397 6955 server_base.cc:1034] running on GCE node
I20260929 11:32:52.970973 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:52.971882 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:52.973078 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790681572973021 us; error 51 us; skew 500 ppm
I20260929 11:32:52.973379 6955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:52 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:52.976539 6955 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:32:52.978041 6955 webserver.cc:460] Webserver started at http://127.6.9.194:44031/ using document root <none> and password file <none>
I20260929 11:32:52.978778 6955 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:52.978895 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:52.979180 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:52.982156 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "4dbd7119d2884717afcaadd99d6ef81c"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "6af2523cbfe540da407faffa85db2bff"
server_key_iv: "785b7c343babb0142c416740a37cc813"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.982957 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "4dbd7119d2884717afcaadd99d6ef81c"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "6af2523cbfe540da407faffa85db2bff"
server_key_iv: "785b7c343babb0142c416740a37cc813"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.988116 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:32:52.991616 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:52.992990 6955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:32:52.993132 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "4dbd7119d2884717afcaadd99d6ef81c"
format_stamp: "Formatted at 2026-09-29 11:32:52 on dist-test-slave-c0l9"
server_key: "6af2523cbfe540da407faffa85db2bff"
server_key_iv: "785b7c343babb0142c416740a37cc813"
server_key_version: "encryptionkey@0"
I20260929 11:32:52.993310 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:53.010795 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:53.015235 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:53.015518 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:53.016520 6955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:53.017976 6955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:53.018054 6955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:53.018158 6955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:53.018208 6955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:53.047925 6955 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:36649
I20260929 11:32:53.047971 7082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:36649 every 8 connection(s)
I20260929 11:32:53.049190 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:32:53.055830 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 6955
I20260929 11:32:53.055991 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:32:53.056463 6183 external_mini_cluster.cc:1428] Setting key 40d8781695cf6af06a5585d0aff101d5
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:53.069413 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.051596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:60189 (local address 127.6.9.254:35307)
0929 11:32:53.051797 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.051803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.052958 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 11:32:53.054718 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.054722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.054801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.054888 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.056325 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.057302 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.059010 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.059477 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.059582 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.063410 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.063415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.063420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.063459 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.066582 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.067226 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.067232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.067235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.067384 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.067731 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.067735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.067739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.068079 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.068215 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.068835 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.069066 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:32:53.070227 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.052021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:35307 (local address 127.6.9.194:60189)
0929 11:32:53.052761 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.052820 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.054485 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.055069 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.055077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.055587 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:53.056167 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.056186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.057454 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.057459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.058860 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.058875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.059704 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.059711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.059993 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.060687 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:53.060719 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:53.063191 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:53.066716 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:53.066723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:53.066735 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:53.067128 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:53.067472 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:53.067474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:53.067477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:53.067643 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:53.068177 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:53.068183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:53.068477 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.068758 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.069150 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":276,"threads_started":1}
I20260929 11:32:53.071509 7083 heartbeater.cc:347] Connected to a master server at 127.6.9.254:35307
I20260929 11:32:53.071827 7083 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:32:53.072733 7083 heartbeater.cc:511] Master 127.6.9.254:35307 requested a full tablet report, sending...
I20260929 11:32:53.074239 6760 ts_manager.cc:194] Registered new tserver with Master: 4dbd7119d2884717afcaadd99d6ef81c (127.6.9.194:36649)
I20260929 11:32:53.075117 6760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:60189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:32:53.096375 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:53.405575 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:53.405857 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:53.405982 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:53.413298 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:53.413429 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:53.413599 7088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:32:53.424211 7088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:35307
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:53.425587 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:53.427162 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:53.445163 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:53.445197 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:53.445147 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:53.447000 7088 server_base.cc:1034] running on GCE node
I20260929 11:32:53.447691 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:53.448714 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:53.449985 7088 hybrid_clock.cc:648] HybridClock initialized: now 1790681573449909 us; error 60 us; skew 500 ppm
I20260929 11:32:53.450337 7088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:53.453508 7088 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:32:53.454942 7088 webserver.cc:460] Webserver started at http://127.6.9.195:45025/ using document root <none> and password file <none>
I20260929 11:32:53.455631 7088 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:53.455741 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:53.456043 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:53.459025 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "0b60d5a2344941ff813b1574b01ed995"
format_stamp: "Formatted at 2026-09-29 11:32:53 on dist-test-slave-c0l9"
server_key: "04e1d9e2a274cf224ea55a5fc7cab557"
server_key_iv: "78d89da8b4ad7d7db039b4ea999668e8"
server_key_version: "encryptionkey@0"
I20260929 11:32:53.459946 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "0b60d5a2344941ff813b1574b01ed995"
format_stamp: "Formatted at 2026-09-29 11:32:53 on dist-test-slave-c0l9"
server_key: "04e1d9e2a274cf224ea55a5fc7cab557"
server_key_iv: "78d89da8b4ad7d7db039b4ea999668e8"
server_key_version: "encryptionkey@0"
I20260929 11:32:53.464913 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:32:53.468117 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:53.469368 7088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:53.469503 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "0b60d5a2344941ff813b1574b01ed995"
format_stamp: "Formatted at 2026-09-29 11:32:53 on dist-test-slave-c0l9"
server_key: "04e1d9e2a274cf224ea55a5fc7cab557"
server_key_iv: "78d89da8b4ad7d7db039b4ea999668e8"
server_key_version: "encryptionkey@0"
I20260929 11:32:53.469673 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:53.482758 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:53.486913 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:53.487208 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:53.488181 7088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:32:53.489709 7088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:53.489789 7088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:53.489920 7088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:53.489987 7088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:53.520607 7088 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:33579
I20260929 11:32:53.520705 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:33579 every 8 connection(s)
I20260929 11:32:53.522027 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:32:53.532052 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7088
I20260929 11:32:53.532233 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:32:53.532682 6183 external_mini_cluster.cc:1428] Setting key 2ecbf3c8885ee508648f7075ede09f7d
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:53.540992 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.524410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:46913 (local address 127.6.9.254:35307)
0929 11:32:53.524591 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.524595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.525775 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 11:32:53.527536 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.527542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.527619 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.527706 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.529169 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.529883 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.531357 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.531797 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.531940 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.535626 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.535632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.535637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.535675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.538225 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.538949 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.538952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.538954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.539093 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.539431 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.539434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.539436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.539693 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.539802 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.540446 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.540651 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:32:53.541873 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.524894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:35307 (local address 127.6.9.195:46913)
0929 11:32:53.525623 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.525665 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.527281 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.527926 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.527934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.528424 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:53.529009 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.529034 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.530019 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.530030 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.531224 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.531237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.532020 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.532037 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.532293 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.532984 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:53.533006 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:53.535410 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:53.538373 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:53.538380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:53.538395 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:53.538845 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:53.539170 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:53.539173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:53.539175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:53.539353 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:53.539781 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:53.539787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:53.540102 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.540323 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.540765 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":281,"threads_started":1}
I20260929 11:32:53.543113 7216 heartbeater.cc:347] Connected to a master server at 127.6.9.254:35307
I20260929 11:32:53.543398 7216 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:53.544078 7216 heartbeater.cc:511] Master 127.6.9.254:35307 requested a full tablet report, sending...
I20260929 11:32:53.545431 6760 ts_manager.cc:194] Registered new tserver with Master: 0b60d5a2344941ff813b1574b01ed995 (127.6.9.195:33579)
I20260929 11:32:53.546295 6760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:46913
I20260929 11:32:53.547822 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:32:53.566418 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.554089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.6.9.254:35307)
0929 11:32:53.554244 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.554248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.554712 (+ 464us) server_negotiation.cc:407] Connection header received
0929 11:32:53.554918 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.554923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.554997 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.555099 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.556512 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.557304 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.558543 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.558983 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.559091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.561341 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.561346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.561349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.561388 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.563940 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.564584 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.564587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.564589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.564775 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.565143 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.565146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.565148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.565465 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.565647 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.566006 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.566183 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:32:53.572824 6760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:32:53.579624 6760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:32:53.601096 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.592002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:33579 (local address 127.0.0.1:34024)
0929 11:32:53.592706 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.592742 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.592928 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.594349 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.594356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.594387 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:53.594730 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.594744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.596628 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.596650 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.598391 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.598403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.600136 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.600142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.600355 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.600404 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.600549 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.600755 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":1085,"threads_started":1}
I20260929 11:32:53.601109 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.592115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.6.9.195:33579)
0929 11:32:53.593481 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.593489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.593520 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:32:53.593611 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.593618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.593834 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.594035 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:53.594930 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.596488 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.598537 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.599931 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.600055 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.600284 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.600589 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.600767 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":1069,"threads_started":1}
I20260929 11:32:53.603264 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.594400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36649 (local address 127.0.0.1:59616)
0929 11:32:53.595265 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.595281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.595378 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.596517 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.596522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.596550 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:53.597007 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.597020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.598948 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.598951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.600719 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.600732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.602496 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.602503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.602675 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.602707 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.602802 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.602923 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":672,"threads_started":1}
I20260929 11:32:53.603338 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.594883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.6.9.194:36649)
0929 11:32:53.595937 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.595945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.595970 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:32:53.596066 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.596074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.596273 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.597223 (+ 950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:53.597426 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.598823 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.600884 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.602286 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.602414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.602596 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.602862 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.603023 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":814,"threads_started":1}
I20260929 11:32:53.603878 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.593826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41297 (local address 127.0.0.1:60760)
0929 11:32:53.594351 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.594371 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.594501 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.596068 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.596072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.596097 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:53.596494 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.596517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.598612 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.598617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.600610 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.600623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.603160 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.603167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.603321 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.603351 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.603417 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.603558 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260929 11:32:53.604708 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.594062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.6.9.193:41297)
0929 11:32:53.594999 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.595016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.595044 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:32:53.595139 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.595146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.595344 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.596744 (+ 1400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:53.596938 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.598445 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.601463 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.602935 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.603949 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.604164 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.604236 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.604379 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":690,"threads_started":1}
I20260929 11:32:53.605544 7151 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12e56d253d42c7b916cb4c46ca3e72 (DEFAULT_TABLE table=test-table [id=fb51ec824d964cfba761e01b65faf4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:53.607004 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12e56d253d42c7b916cb4c46ca3e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:53.606681 7018 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12e56d253d42c7b916cb4c46ca3e72 (DEFAULT_TABLE table=test-table [id=fb51ec824d964cfba761e01b65faf4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:53.607975 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12e56d253d42c7b916cb4c46ca3e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:53.608088 6884 tablet_service.cc:1467] Processing CreateTablet for tablet 0a12e56d253d42c7b916cb4c46ca3e72 (DEFAULT_TABLE table=test-table [id=fb51ec824d964cfba761e01b65faf4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:53.609386 6884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a12e56d253d42c7b916cb4c46ca3e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:53.617880 7234 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Bootstrap starting.
I20260929 11:32:53.618619 6949 heartbeater.cc:503] Master 127.6.9.254:35307 was elected leader, sending a full tablet report...
I20260929 11:32:53.621937 7235 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Bootstrap starting.
I20260929 11:32:53.622473 7234 tablet_bootstrap.cc:654] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:53.623281 7236 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Bootstrap starting.
I20260929 11:32:53.624748 7234 log.cc:824] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:53.625547 7235 tablet_bootstrap.cc:654] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:53.626405 7236 tablet_bootstrap.cc:654] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:53.626590 7235 log.cc:824] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:53.627449 7236 log.cc:824] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:53.627967 7234 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: No bootstrap required, opened a new log
I20260929 11:32:53.628284 7234 ts_tablet_manager.cc:1397] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 11:32:53.629808 7235 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: No bootstrap required, opened a new log
I20260929 11:32:53.630064 7235 ts_tablet_manager.cc:1397] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 11:32:53.635524 7236 tablet_bootstrap.cc:492] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: No bootstrap required, opened a new log
I20260929 11:32:53.635790 7236 ts_tablet_manager.cc:1397] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 11:32:53.639915 7234 raft_consensus.cc:348] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.640403 7234 raft_consensus.cc:374] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:53.640540 7234 raft_consensus.cc:729] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b60d5a2344941ff813b1574b01ed995, State: Initialized, Role: FOLLOWER
I20260929 11:32:53.640766 7235 raft_consensus.cc:348] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.641299 7235 raft_consensus.cc:374] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:53.641398 7235 raft_consensus.cc:729] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbd7119d2884717afcaadd99d6ef81c, State: Initialized, Role: FOLLOWER
I20260929 11:32:53.641251 7234 consensus_queue.cc:260] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.641986 7235 consensus_queue.cc:260] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.643217 7083 heartbeater.cc:503] Master 127.6.9.254:35307 was elected leader, sending a full tablet report...
I20260929 11:32:53.643918 7235 ts_tablet_manager.cc:1428] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 11:32:53.645190 7236 raft_consensus.cc:348] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.645607 7236 raft_consensus.cc:374] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:53.645709 7236 raft_consensus.cc:729] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ad71d80f784ee6aaa47d23c70873ea, State: Initialized, Role: FOLLOWER
I20260929 11:32:53.646291 7236 consensus_queue.cc:260] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.647939 7236 ts_tablet_manager.cc:1428] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 11:32:53.648972 7216 heartbeater.cc:503] Master 127.6.9.254:35307 was elected leader, sending a full tablet report...
I20260929 11:32:53.650669 7234 ts_tablet_manager.cc:1428] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260929 11:32:53.686385 7241 raft_consensus.cc:482] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:32:53.686699 7241 raft_consensus.cc:504] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.688405 7241 leader_election.cc:288] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b60d5a2344941ff813b1574b01ed995 (127.6.9.195:33579), 25ad71d80f784ee6aaa47d23c70873ea (127.6.9.193:41297)
I20260929 11:32:53.695740 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.688769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:33579 (local address 127.6.9.194:47991)
0929 11:32:53.689467 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.689491 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.689650 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.690230 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.690234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.690263 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:53.690611 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.690620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.691687 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.691692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.693718 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.693730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.694937 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.694944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.695076 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.695099 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.695182 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.695311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":23,"thread_start_us":308,"threads_started":1}
I20260929 11:32:53.695804 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.688891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:47991 (local address 127.6.9.195:33579)
0929 11:32:53.689063 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.689068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.689577 (+ 509us) server_negotiation.cc:407] Connection header received
0929 11:32:53.689847 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.689855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.689941 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.690009 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:53.690757 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.691560 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.693869 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.694711 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.694828 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.694933 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.695224 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.695354 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:32:53.696143 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.689819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59371 (local address 127.6.9.193:41297)
0929 11:32:53.690006 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.690011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.690546 (+ 535us) server_negotiation.cc:407] Connection header received
0929 11:32:53.690733 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.690737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.690794 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.690868 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:53.691469 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.692250 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.694368 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.695171 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.695294 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.695395 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.695781 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.695910 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:32:53.696316 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.689639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41297 (local address 127.6.9.194:59371)
0929 11:32:53.690436 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:53.690451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:53.690601 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:53.690891 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:53.690895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:53.690918 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:53.691325 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.691333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.692391 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.692394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.694215 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:53.694228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.695412 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.695419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:53.695568 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.695604 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:32:53.695747 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 11:32:53.695877 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":23,"thread_start_us":489,"threads_started":1}
I20260929 11:32:53.697191 7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12e56d253d42c7b916cb4c46ca3e72" candidate_uuid: "4dbd7119d2884717afcaadd99d6ef81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b60d5a2344941ff813b1574b01ed995" is_pre_election: true
I20260929 11:32:53.697511 6904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12e56d253d42c7b916cb4c46ca3e72" candidate_uuid: "4dbd7119d2884717afcaadd99d6ef81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ad71d80f784ee6aaa47d23c70873ea" is_pre_election: true
I20260929 11:32:53.698091 7171 raft_consensus.cc:2454] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbd7119d2884717afcaadd99d6ef81c in term 0.
I20260929 11:32:53.698752 6973 leader_election.cc:302] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b60d5a2344941ff813b1574b01ed995, 4dbd7119d2884717afcaadd99d6ef81c; no voters:
I20260929 11:32:53.699102 7241 raft_consensus.cc:2781] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:32:53.699245 7241 raft_consensus.cc:482] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:32:53.699342 7241 raft_consensus.cc:3037] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:53.700052 6904 raft_consensus.cc:2454] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbd7119d2884717afcaadd99d6ef81c in term 0.
I20260929 11:32:53.701679 7241 raft_consensus.cc:504] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.702229 7241 leader_election.cc:288] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [CANDIDATE]: Term 1 election: Requested vote from peers 0b60d5a2344941ff813b1574b01ed995 (127.6.9.195:33579), 25ad71d80f784ee6aaa47d23c70873ea (127.6.9.193:41297)
I20260929 11:32:53.702742 7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12e56d253d42c7b916cb4c46ca3e72" candidate_uuid: "4dbd7119d2884717afcaadd99d6ef81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b60d5a2344941ff813b1574b01ed995"
I20260929 11:32:53.702795 6904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a12e56d253d42c7b916cb4c46ca3e72" candidate_uuid: "4dbd7119d2884717afcaadd99d6ef81c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ad71d80f784ee6aaa47d23c70873ea"
I20260929 11:32:53.702970 7171 raft_consensus.cc:3037] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:53.702972 6904 raft_consensus.cc:3037] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:53.706125 6904 raft_consensus.cc:2454] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbd7119d2884717afcaadd99d6ef81c in term 1.
I20260929 11:32:53.706123 7171 raft_consensus.cc:2454] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbd7119d2884717afcaadd99d6ef81c in term 1.
I20260929 11:32:53.706534 6973 leader_election.cc:302] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ad71d80f784ee6aaa47d23c70873ea, 4dbd7119d2884717afcaadd99d6ef81c; no voters:
I20260929 11:32:53.706822 7241 raft_consensus.cc:2781] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:53.707253 7241 raft_consensus.cc:686] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [term 1 LEADER]: Becoming Leader. State: Replica: 4dbd7119d2884717afcaadd99d6ef81c, State: Running, Role: LEADER
I20260929 11:32:53.707723 7241 consensus_queue.cc:237] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } }
I20260929 11:32:53.712476 6759 catalog_manager.cc:5495] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbd7119d2884717afcaadd99d6ef81c (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4dbd7119d2884717afcaadd99d6ef81c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b60d5a2344941ff813b1574b01ed995" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 33579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ad71d80f784ee6aaa47d23c70873ea" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbd7119d2884717afcaadd99d6ef81c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36649 } health_report { overall_health: HEALTHY } } }
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:32:53.733698 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.719539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.6.9.193:41297)
0929 11:32:53.719688 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.719693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.720150 (+ 457us) server_negotiation.cc:407] Connection header received
0929 11:32:53.720314 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.720319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.720399 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.720488 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.722020 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.722796 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.723998 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.724413 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.724527 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.728335 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.728349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.728366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.728397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.731059 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.731706 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.731713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.731715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.731834 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.732197 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.732201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.732203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.732637 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.732901 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.733161 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.733347 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
W20260929 11:32:53.773584 7217 tablet.cc:2369] T 0a12e56d253d42c7b916cb4c46ca3e72 P 0b60d5a2344941ff813b1574b01ed995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:53.777676 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.762581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.6.9.193:41297)
0929 11:32:53.762754 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.762760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.763114 (+ 354us) server_negotiation.cc:407] Connection header received
0929 11:32:53.763291 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.763295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.763362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.763446 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.764892 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.765775 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.767070 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.767483 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.767598 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.771760 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.771766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.771769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.771811 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.774793 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.775531 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.775536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.775538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.775706 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.776112 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.776116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.776118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.776447 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.776831 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.777078 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.777311 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 11:32:53.801098 7084 tablet.cc:2369] T 0a12e56d253d42c7b916cb4c46ca3e72 P 4dbd7119d2884717afcaadd99d6ef81c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:32:53.820451 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.806550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.6.9.193:41297)
0929 11:32:53.806815 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.806822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.807022 (+ 200us) server_negotiation.cc:407] Connection header received
0929 11:32:53.807201 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.807205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.807273 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.807350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.808790 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.809476 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.810727 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.811182 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.811279 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.815457 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.815463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.815466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.815505 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.817982 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.818646 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.818650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.818653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.818788 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.819081 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.819083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.819089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.819401 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.819610 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.819883 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.820080 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260929 11:32:53.822273 6864 tablet_service.cc:2908] Rejecting scan request for tablet 0a12e56d253d42c7b916cb4c46ca3e72: Uninitialized: clean time has not yet been initialized
I20260929 11:32:53.825572 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6821
I20260929 11:32:53.838748 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.826794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.6.9.193:41297)
0929 11:32:53.826921 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.826924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.827290 (+ 366us) server_negotiation.cc:407] Connection header received
0929 11:32:53.827433 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.827437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.827512 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.827586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.828749 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.829403 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.830595 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.831012 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.831121 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.833362 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.833366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.833370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.833399 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.836135 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.836811 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.836815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.836817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.836965 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.837397 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.837400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.837403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.837711 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.837944 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.838294 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.838468 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 11:32:53.839372 6944 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 11:32:53.926414 6950 tablet.cc:2369] T 0a12e56d253d42c7b916cb4c46ca3e72 P 25ad71d80f784ee6aaa47d23c70873ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:53.927032 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6821
I20260929 11:32:53.940037 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6955
Sep 29 11:32:53 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:32:53.953743 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:53.940873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.6.9.194:36649)
0929 11:32:53.941047 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:53.941051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:53.941087 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:32:53.941141 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:53.941145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:53.941205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:53.941286 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:53.942490 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.943222 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.944550 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:53.945040 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:53.945138 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:53.948748 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:53.948762 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:53.948777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:53.948811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:53.951328 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.951955 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.951962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.951965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.952125 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:53.952449 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:53.952453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:53.952455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:53.952853 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:53.953106 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:53.953270 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 11:32:53.953434 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:32:53.954391 7078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:54.052701 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6955
I20260929 11:32:54.065634 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7088
Sep 29 11:32:54 dist-test-slave-c0l9 krb5kdc[6715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:32:54.079619 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:54.066384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.6.9.195:33579)
0929 11:32:54.066575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:54.066580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:54.066599 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:32:54.066765 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:54.066769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:54.066833 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:54.066916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:54.068120 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:54.068885 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:54.070107 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:54.070513 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:54.070617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:54.074542 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:54.074554 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:54.074572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:54.074608 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:54.077094 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:54.077690 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:54.077697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:54.077699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:54.077837 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:54.078192 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:54.078196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:54.078198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:54.078532 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:54.078794 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:54.079077 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 11:32:54.079275 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:32:54.080201 7211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:54.169096 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7088
I20260929 11:32:54.181633 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 6728
I20260929 11:32:54.182762 6790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:54.289716 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 6728
2026-09-29T11:32:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3851 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:54 dist-test-slave-c0l9 krb5kdc[7264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:54 dist-test-slave-c0l9 krb5kdc[7264](info): set up 2 sockets
Sep 29 11:32:54 dist-test-slave-c0l9 krb5kdc[7264](info): commencing operation
krb5kdc: starting...
W20260929 11:32:55.280344 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:32:55 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:32:55.448778 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44777
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:55.752847 7277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:55.753145 7277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:55.753242 7277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:55.760324 7277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:55.760423 7277 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:32:55.760462 7277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:55.760521 7277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:55.760571 7277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:55.760637 7277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:55.770539 7277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:44777
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:55.771785 7277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:55.773114 7277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:55.784860 7282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:55.784888 7283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:55.784935 7285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:55.785787 7277 server_base.cc:1034] running on GCE node
I20260929 11:32:55.786568 7277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:55.787982 7277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:55.789204 7277 hybrid_clock.cc:648] HybridClock initialized: now 1790681575789145 us; error 37 us; skew 500 ppm
I20260929 11:32:55.789503 7277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:55 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:55.792496 7277 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:32:55.794083 7277 webserver.cc:460] Webserver started at http://127.6.9.254:38819/ using document root <none> and password file <none>
I20260929 11:32:55.794772 7277 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:55.794862 7277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:55.795152 7277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:55.798278 7277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "32f6e4e5ec3141eda43ba26ed1a2293c"
format_stamp: "Formatted at 2026-09-29 11:32:55 on dist-test-slave-c0l9"
server_key: "23b862fb932167956c5519ae77c3ef39"
server_key_iv: "ff91750974621625da4c9e41ee6717c1"
server_key_version: "encryptionkey@0"
I20260929 11:32:55.799108 7277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "32f6e4e5ec3141eda43ba26ed1a2293c"
format_stamp: "Formatted at 2026-09-29 11:32:55 on dist-test-slave-c0l9"
server_key: "23b862fb932167956c5519ae77c3ef39"
server_key_iv: "ff91750974621625da4c9e41ee6717c1"
server_key_version: "encryptionkey@0"
I20260929 11:32:55.804879 7277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:32:55.808518 7292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:55.809976 7277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:55.810146 7277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "32f6e4e5ec3141eda43ba26ed1a2293c"
format_stamp: "Formatted at 2026-09-29 11:32:55 on dist-test-slave-c0l9"
server_key: "23b862fb932167956c5519ae77c3ef39"
server_key_iv: "ff91750974621625da4c9e41ee6717c1"
server_key_version: "encryptionkey@0"
I20260929 11:32:55.810345 7277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:55.824856 7277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:55.829218 7277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:55.829452 7277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:55.849565 7277 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:44777
I20260929 11:32:55.849586 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:44777 every 8 connection(s)
I20260929 11:32:55.850872 7277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:55.854736 7354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:55.854795 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7277
I20260929 11:32:55.855012 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:32:55.855906 6183 external_mini_cluster.cc:1428] Setting key 099248d1b90b4dbf467f33845de9c513
I20260929 11:32:55.868989 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Bootstrap starting.
Sep 29 11:32:55 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:55.872965 7354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:55.874249 7354 log.cc:824] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:55.877385 7354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: No bootstrap required, opened a new log
I20260929 11:32:55.880015 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:55.858166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.6.9.254:44777)
0929 11:32:55.858956 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:55.858968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:55.859009 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:32:55.861187 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:55.861210 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:55.861608 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:55.861940 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:55.863839 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:55.865204 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:55.867002 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:55.867669 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:55.867793 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:55.872644 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:55.872653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:55.872694 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:55.872739 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:55.875855 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:55.876565 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:55.876572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:55.876578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:55.876814 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:55.877188 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:55.877194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:55.877197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:55.877792 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:55.878081 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:55.878520 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 11:32:55.878954 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":373,"threads_started":1}
I20260929 11:32:55.885969 7354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } }
I20260929 11:32:55.886404 7354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:55.886569 7354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f6e4e5ec3141eda43ba26ed1a2293c, State: Initialized, Role: FOLLOWER
I20260929 11:32:55.887179 7354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } }
I20260929 11:32:55.887409 7354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:55.887545 7354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:55.887698 7354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:55.890080 7354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } }
I20260929 11:32:55.890597 7354 leader_election.cc:302] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32f6e4e5ec3141eda43ba26ed1a2293c; no voters:
I20260929 11:32:55.891208 7354 leader_election.cc:288] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:55.891319 7359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:55.891961 7359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [term 1 LEADER]: Becoming Leader. State: Replica: 32f6e4e5ec3141eda43ba26ed1a2293c, State: Running, Role: LEADER
I20260929 11:32:55.892485 7359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } }
I20260929 11:32:55.893381 7354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:55.895519 7361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32f6e4e5ec3141eda43ba26ed1a2293c. Latest consensus state: current_term: 1 leader_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } } }
I20260929 11:32:55.895845 7361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:55.895385 7360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f6e4e5ec3141eda43ba26ed1a2293c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44777 } } }
I20260929 11:32:55.896226 7360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:55.897783 7366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:55.903090 7366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:55.912879 7366 catalog_manager.cc:1347] Generated new cluster ID: 7de003e56b634455a7c5fb5a22396683
I20260929 11:32:55.912995 7366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:55.928153 7366 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:55.929529 7366 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:55.937863 7366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Generated new TSK 0
I20260929 11:32:55.938684 7366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:32:55.977895 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44777
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:56.290783 7380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:56.291067 7380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:56.291162 7380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:56.298641 7380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:56.298774 7380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:56.298870 7380 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:32:56.298951 7380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:32:56.308791 7380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:56.310014 7380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:56.311247 7380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:56.323648 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:56.323604 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:56.323577 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:56.324391 7380 server_base.cc:1034] running on GCE node
I20260929 11:32:56.324995 7380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:56.325830 7380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:56.327040 7380 hybrid_clock.cc:648] HybridClock initialized: now 1790681576326972 us; error 46 us; skew 500 ppm
I20260929 11:32:56.327323 7380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:56 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681576, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:56.330284 7380 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:32:56.331617 7380 webserver.cc:460] Webserver started at http://127.6.9.193:44073/ using document root <none> and password file <none>
I20260929 11:32:56.332288 7380 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:56.332388 7380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:56.332739 7380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:56.335614 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "ba87fc549df647c4a125ce0087fb24d9"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "04970913f54fc18c8b7876f84cd28279"
server_key_iv: "4fdc0f168ffdfc4e483662fa4485eabc"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.336400 7380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "ba87fc549df647c4a125ce0087fb24d9"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "04970913f54fc18c8b7876f84cd28279"
server_key_iv: "4fdc0f168ffdfc4e483662fa4485eabc"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.341468 7380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:32:56.344836 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:56.346144 7380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:32:56.346311 7380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "ba87fc549df647c4a125ce0087fb24d9"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "04970913f54fc18c8b7876f84cd28279"
server_key_iv: "4fdc0f168ffdfc4e483662fa4485eabc"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.346506 7380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:56.367726 7380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:56.371833 7380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:56.372097 7380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:56.374686 7380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:56.374774 7380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:56.374888 7380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:56.374945 7380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:32:56 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681576, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:56.397279 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.375383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:48299 (local address 127.6.9.254:44777)
0929 11:32:56.375565 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:56.375571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:56.376955 (+ 1384us) server_negotiation.cc:407] Connection header received
0929 11:32:56.379117 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:56.379124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:56.379218 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:56.379297 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:56.381762 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.382889 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.385169 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.385846 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.385995 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.391054 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:56.391060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:56.391063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:56.391099 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:56.394114 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.394952 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.394958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.394962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.395093 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.395524 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.395528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.395530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.395814 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:56.395948 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:56.396705 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 11:32:56.396927 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:32:56.398447 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.375884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.193:48299)
0929 11:32:56.376770 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:56.376816 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:56.378715 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:56.379858 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:56.379873 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:56.380715 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:56.381537 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.381562 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.383059 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.383064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.385015 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.385034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.386146 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.386153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.386483 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.387432 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:56.387478 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:56.390777 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:56.394281 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.394298 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.394315 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.394821 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.395207 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.395212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.395216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.395409 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.395936 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:56.395943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:56.396314 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 11:32:56.396552 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 11:32:56.397120 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":342,"threads_started":1}
I20260929 11:32:56.418623 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.403680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.193:41199)
0929 11:32:56.404301 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:56.404317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:56.404449 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:56.404805 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:56.404810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:56.405552 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:56.406022 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.406040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.407481 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.407487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.409077 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.409094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.410093 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.410099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.410296 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.411092 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:56.411110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:56.412222 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:56.415676 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.415680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.415683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.416247 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.416602 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.416607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.416611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.416784 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.417342 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:56.417347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:56.417590 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 11:32:56.417790 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 11:32:56.418124 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":363,"threads_started":1}
I20260929 11:32:56.419359 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.403639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:41199 (local address 127.6.9.254:44777)
0929 11:32:56.403784 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:56.403788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:56.404395 (+ 607us) server_negotiation.cc:407] Connection header received
0929 11:32:56.404557 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:56.404563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:56.404632 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:56.404746 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:56.406231 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.407322 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.409233 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.409897 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.410032 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.412424 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:56.412429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:56.412432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:56.412464 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:56.415487 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.416364 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.416369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.416373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.416499 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.416886 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.416892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.416896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.417245 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:56.417382 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:56.418960 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0929 11:32:56.419138 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 11:32:56.421912 7380 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:40267
I20260929 11:32:56.421931 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:40267 every 8 connection(s)
I20260929 11:32:56.423416 7380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:32:56.423909 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7380
I20260929 11:32:56.424104 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:32:56.424922 6183 external_mini_cluster.cc:1428] Setting key 2ebd2339df65eba6a1525cd266f8a853
I20260929 11:32:56.425405 7515 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44777
I20260929 11:32:56.425618 7515 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:56.426370 7515 heartbeater.cc:511] Master 127.6.9.254:44777 requested a full tablet report, sending...
I20260929 11:32:56.428504 7309 ts_manager.cc:194] Registered new tserver with Master: ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193:40267)
I20260929 11:32:56.430425 7309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:48299
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:32:56.461831 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44777
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:56.767588 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:56.767889 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:56.768013 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:56.775303 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:56.775439 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:56.775568 7519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:32:56.775677 7519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:32:56.785480 7519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:56.786770 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:56.788058 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:56.800437 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:56.800478 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:56.800506 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:56.801573 7519 server_base.cc:1034] running on GCE node
I20260929 11:32:56.802174 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:56.803097 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:56.804328 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790681576804258 us; error 50 us; skew 500 ppm
I20260929 11:32:56.804612 7519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:56 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681576, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:56.807647 7519 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:32:56.809116 7519 webserver.cc:460] Webserver started at http://127.6.9.194:35465/ using document root <none> and password file <none>
I20260929 11:32:56.809779 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:56.809870 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:56.810168 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:56.813088 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "e669905e605645afa81efd36d1c98bd3"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "54b43aa65e14d2b8ee7070eb9edb2d7b"
server_key_iv: "5ca657744c3e4f5df3b58090bbe72bf6"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.813865 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "e669905e605645afa81efd36d1c98bd3"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "54b43aa65e14d2b8ee7070eb9edb2d7b"
server_key_iv: "5ca657744c3e4f5df3b58090bbe72bf6"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.818989 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:32:56.822451 7534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:56.823725 7519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:32:56.823874 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "e669905e605645afa81efd36d1c98bd3"
format_stamp: "Formatted at 2026-09-29 11:32:56 on dist-test-slave-c0l9"
server_key: "54b43aa65e14d2b8ee7070eb9edb2d7b"
server_key_iv: "5ca657744c3e4f5df3b58090bbe72bf6"
server_key_version: "encryptionkey@0"
I20260929 11:32:56.824092 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:56.835232 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:56.838944 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:56.839180 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:56.841683 7519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:56.841764 7519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:56.841925 7519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:56.841993 7519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:32:56 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681576, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:56.866393 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.842061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:54653 (local address 127.6.9.254:44777)
0929 11:32:56.842269 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:56.842277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:56.843624 (+ 1347us) server_negotiation.cc:407] Connection header received
0929 11:32:56.846306 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:56.846313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:56.846397 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:56.846500 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:56.848752 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.849926 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.852947 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.853617 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.853746 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.858879 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:56.858886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:56.858892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:56.858937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:56.862419 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.863417 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.863424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.863428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.863566 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.864122 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.864128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.864132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.864572 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:56.864765 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:56.865609 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 11:32:56.865883 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:32:56.867389 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.842616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.194:54653)
0929 11:32:56.843431 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:56.843480 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:56.845994 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:56.846831 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:56.846842 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:56.847626 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:56.848463 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.848493 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.850089 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.850096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.852750 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.852769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.853869 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.853877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.854181 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.855082 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:56.855116 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:56.858506 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:56.862587 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.862599 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.862620 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.863267 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.863700 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.863705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.863709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.863971 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.864740 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:56.864748 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:56.865202 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 11:32:56.865480 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 11:32:56.866023 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":328,"threads_started":1}
I20260929 11:32:56.887725 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.872627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:57715 (local address 127.6.9.254:44777)
0929 11:32:56.872848 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:56.872855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:56.873720 (+ 865us) server_negotiation.cc:407] Connection header received
0929 11:32:56.873939 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:56.873945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:56.874029 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:56.874126 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:56.875373 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.876133 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.877917 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.878476 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.878599 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.881198 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:56.881204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:56.881209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:56.881252 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:56.884401 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.885405 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.885411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.885415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.885619 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:56.886125 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:56.886131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:56.886135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:56.886475 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:56.886604 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:56.887260 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 11:32:56.887454 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:32:56.888008 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:56.872916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.194:57715)
0929 11:32:56.873600 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:56.873621 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:56.873788 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:56.874180 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:56.874185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:56.874794 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:56.875229 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.875241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.876267 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.876272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.877771 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:56.877791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:56.878900 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:56.878905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:56.879101 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:56.879871 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:56.879894 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:56.880935 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:56.884617 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.884626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.884630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.885264 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.885761 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:56.885766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:56.885773 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:56.885972 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:56.886700 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:56.886705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:56.886942 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 11:32:56.887113 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 11:32:56.887496 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":368,"threads_started":1}
I20260929 11:32:56.898453 7519 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:45433
I20260929 11:32:56.898491 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:45433 every 8 connection(s)
I20260929 11:32:56.899761 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:32:56.901782 7654 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44777
I20260929 11:32:56.901978 7654 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:56.902576 7654 heartbeater.cc:511] Master 127.6.9.254:44777 requested a full tablet report, sending...
I20260929 11:32:56.903865 7309 ts_manager.cc:194] Registered new tserver with Master: e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433)
I20260929 11:32:56.904696 7309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:54653
I20260929 11:32:56.907228 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7519
I20260929 11:32:56.907413 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:32:56.907843 6183 external_mini_cluster.cc:1428] Setting key 7e9e108c743ef892c45a5ac1b4f10751
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:32:56.943905 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44777
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:57.253345 7658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:57.253614 7658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:57.253715 7658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:57.260768 7658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:57.260872 7658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:57.260962 7658 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:32:57.261039 7658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:32:57.270695 7658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:44777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:57.271893 7658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:57.273200 7658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:57.285498 7664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:57.285537 7663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:57.285573 7666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:57.286281 7658 server_base.cc:1034] running on GCE node
I20260929 11:32:57.286835 7658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:57.287669 7658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:57.288869 7658 hybrid_clock.cc:648] HybridClock initialized: now 1790681577288807 us; error 33 us; skew 500 ppm
I20260929 11:32:57.289149 7658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:57 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:57.296643 7658 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:32:57.298214 7658 webserver.cc:460] Webserver started at http://127.6.9.195:45429/ using document root <none> and password file <none>
I20260929 11:32:57.298951 7658 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:57.299047 7658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:57.299350 7658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:57.302392 7658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
format_stamp: "Formatted at 2026-09-29 11:32:57 on dist-test-slave-c0l9"
server_key: "7bcf6874f35c1a5235325ba7cd3f987a"
server_key_iv: "5abace501d072b1f92ad8b43ac81b526"
server_key_version: "encryptionkey@0"
I20260929 11:32:57.303143 7658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
format_stamp: "Formatted at 2026-09-29 11:32:57 on dist-test-slave-c0l9"
server_key: "7bcf6874f35c1a5235325ba7cd3f987a"
server_key_iv: "5abace501d072b1f92ad8b43ac81b526"
server_key_version: "encryptionkey@0"
I20260929 11:32:57.308074 7658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:32:57.311381 7673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:57.312629 7658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:32:57.312791 7658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
format_stamp: "Formatted at 2026-09-29 11:32:57 on dist-test-slave-c0l9"
server_key: "7bcf6874f35c1a5235325ba7cd3f987a"
server_key_iv: "5abace501d072b1f92ad8b43ac81b526"
server_key_version: "encryptionkey@0"
I20260929 11:32:57.312961 7658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:57.328814 7658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:57.332749 7658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:57.332953 7658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:57.335500 7658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:32:57.335608 7658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:57.335762 7658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:32:57.335879 7658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:32:57 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:57.356693 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.335724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:53051 (local address 127.6.9.254:44777)
0929 11:32:57.335919 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.335926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.337139 (+ 1213us) server_negotiation.cc:407] Connection header received
0929 11:32:57.339115 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.339122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.339212 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.339297 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:57.341291 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.342223 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.344723 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.345379 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.345525 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.349860 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:57.349867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:57.349872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:57.349934 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:57.353206 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.354185 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.354192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.354196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.354378 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.354784 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.354789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.354793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.355180 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:57.355335 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.355998 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.356281 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:32:57.357563 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.336110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.195:53051)
0929 11:32:57.336970 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.337018 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.338843 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.339526 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.339533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.340217 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:57.341057 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.341086 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.342388 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.342392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.344532 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.344545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.345640 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.345647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.345839 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.346553 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:57.346589 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:57.349597 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:57.353333 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.353346 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.353363 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.354022 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.354486 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.354490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.354492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.354673 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.355289 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:57.355295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:57.355629 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.355864 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.356281 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":365,"threads_started":1}
I20260929 11:32:57.375533 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.362666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:49731 (local address 127.6.9.254:44777)
0929 11:32:57.362816 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.362821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.363558 (+ 737us) server_negotiation.cc:407] Connection header received
0929 11:32:57.363724 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.363727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.363787 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.363864 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:57.364783 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.365681 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.367520 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.368040 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.368151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.370102 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:57.370108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:57.370111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:57.370149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:57.372625 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.373406 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.373412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.373415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.373607 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.374024 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.374029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.374032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.374385 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:57.374516 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.374966 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.375165 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:32:57.375660 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.362720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.6.9.195:49731)
0929 11:32:57.363464 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.363480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.363609 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.363900 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.363904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.364342 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:57.364619 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.364628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.365859 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.365863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.367316 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.367333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.368260 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.368268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.368426 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.369222 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:57.369244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:57.369936 (+ 692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:57.372764 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.372769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.372773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.373236 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.373757 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.373765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.373769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.373910 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.374495 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:57.374499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:57.374700 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.374853 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.375214 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":396,"threads_started":1}
I20260929 11:32:57.383100 7658 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:35051
I20260929 11:32:57.383117 7792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:35051 every 8 connection(s)
I20260929 11:32:57.384253 7658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:32:57.386053 7793 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44777
I20260929 11:32:57.386274 7793 heartbeater.cc:464] Registering TS with master...
I20260929 11:32:57.386866 7793 heartbeater.cc:511] Master 127.6.9.254:44777 requested a full tablet report, sending...
I20260929 11:32:57.388065 7309 ts_manager.cc:194] Registered new tserver with Master: 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:57.388895 7309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:53051
I20260929 11:32:57.389561 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7658
I20260929 11:32:57.389737 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:32:57.390143 6183 external_mini_cluster.cc:1428] Setting key 51e5425ed97630781f18718de715b250
I20260929 11:32:57.392746 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:32:57 dist-test-slave-c0l9 krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:32:57 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:57.433439 7515 heartbeater.cc:503] Master 127.6.9.254:44777 was elected leader, sending a full tablet report...
I20260929 11:32:57.437848 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.422409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.6.9.254:44777)
0929 11:32:57.422603 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.422609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.422815 (+ 206us) server_negotiation.cc:407] Connection header received
0929 11:32:57.422994 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.422997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.423073 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.423163 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:57.424994 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.425775 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.427463 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.427931 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.428025 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.431645 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:57.431650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:57.431653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:57.431691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:57.434844 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.435739 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.435745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.435749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.435917 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.436345 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.436350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.436354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.436757 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:57.437000 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.437302 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.437532 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:32:57.442721 7309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:32:57.449462 7309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:32:57.487640 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.462515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.6.9.195:35051)
0929 11:32:57.478249 (+ 15734us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.478261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.479382 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 11:32:57.479491 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.479498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.479717 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.479918 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:57.481040 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482618 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.485654 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.486563 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.486675 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.486906 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.487159 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.487327 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15529,"thread_start_us":15323,"threads_started":1}
I20260929 11:32:57.487709 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.461890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:35051 (local address 127.0.0.1:47596)
0929 11:32:57.479131 (+ 17241us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.479149 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.479262 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.480282 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.480288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.480310 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:57.480823 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.480835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482760 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.482765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.485528 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.485537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.486738 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.486745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.486924 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.486985 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.487095 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.487279 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17114,"mutex_wait_us":407,"thread_start_us":16558,"threads_started":1}
I20260929 11:32:57.488252 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.461256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:45433 (local address 127.0.0.1:49542)
0929 11:32:57.475374 (+ 14118us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.475436 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.475667 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.481097 (+ 5430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.481100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.481118 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:57.481414 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.481421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482629 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.482632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.485339 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.485353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.487303 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.487310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.487526 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.487557 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.487640 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.487764 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13820,"mutex_wait_us":102,"thread_start_us":14614,"threads_started":1}
I20260929 11:32:57.488786 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.462071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.6.9.193:40267)
0929 11:32:57.473620 (+ 11549us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.473631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.479105 (+ 5474us) server_negotiation.cc:407] Connection header received
0929 11:32:57.479237 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.479246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.479518 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.479771 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:57.480920 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482521 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.485201 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.486639 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.486778 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.486983 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.488266 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.488432 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11374,"thread_start_us":11238,"threads_started":1}
I20260929 11:32:57.491449 7723 tablet_service.cc:1467] Processing CreateTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 (DEFAULT_TABLE table=test-table [id=26c37479d4174d5cac9df6af7b9035bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.489430 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.461481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:40267 (local address 127.0.0.1:54048)
0929 11:32:57.478731 (+ 17250us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.478755 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.478931 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.480048 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.480058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.480090 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:57.480650 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.480687 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482735 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.482740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.485043 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.485061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.487144 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.487152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.487382 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.487435 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.487518 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.489055 (+ 1537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17115,"mutex_wait_us":134,"thread_start_us":17393,"threads_started":1}
I20260929 11:32:57.492107 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.461365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.6.9.194:45433)
0929 11:32:57.479468 (+ 18103us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.479474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.479496 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:32:57.479574 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.479579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.480731 (+ 1152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.480923 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:57.481564 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.482499 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.485504 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.486858 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.486982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.487187 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.488588 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.488784 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17972,"thread_start_us":17849,"threads_started":1}
I20260929 11:32:57.492908 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a194322979e4ed98e4d6b035c86d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.495575 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 (DEFAULT_TABLE table=test-table [id=26c37479d4174d5cac9df6af7b9035bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.495566 7584 tablet_service.cc:1467] Processing CreateTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 (DEFAULT_TABLE table=test-table [id=26c37479d4174d5cac9df6af7b9035bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.496917 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a194322979e4ed98e4d6b035c86d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.496934 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a194322979e4ed98e4d6b035c86d0c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.504627 7812 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Bootstrap starting.
I20260929 11:32:57.508463 7812 tablet_bootstrap.cc:654] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.509498 7812 log.cc:824] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:57.511559 7814 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Bootstrap starting.
I20260929 11:32:57.512322 7815 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Bootstrap starting.
I20260929 11:32:57.512390 7812 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: No bootstrap required, opened a new log
I20260929 11:32:57.512722 7812 ts_tablet_manager.cc:1397] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 11:32:57.515800 7815 tablet_bootstrap.cc:654] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.515890 7814 tablet_bootstrap.cc:654] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.517141 7814 log.cc:824] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:57.517165 7815 log.cc:824] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:57.520315 7815 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: No bootstrap required, opened a new log
I20260929 11:32:57.520594 7815 ts_tablet_manager.cc:1397] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260929 11:32:57.521297 7814 tablet_bootstrap.cc:492] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: No bootstrap required, opened a new log
I20260929 11:32:57.521530 7814 ts_tablet_manager.cc:1397] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 11:32:57.522754 7812 raft_consensus.cc:348] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.523191 7812 raft_consensus.cc:374] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.523335 7812 raft_consensus.cc:729] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc8a453e0644cfa3eb33f36cf0b3af, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.523979 7812 consensus_queue.cc:260] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.525141 7793 heartbeater.cc:503] Master 127.6.9.254:44777 was elected leader, sending a full tablet report...
I20260929 11:32:57.525784 7812 ts_tablet_manager.cc:1428] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260929 11:32:57.529760 7814 raft_consensus.cc:348] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.530197 7814 raft_consensus.cc:374] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.530311 7814 raft_consensus.cc:729] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.530872 7814 consensus_queue.cc:260] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.531365 7815 raft_consensus.cc:348] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.531798 7815 raft_consensus.cc:374] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.531876 7815 raft_consensus.cc:729] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e669905e605645afa81efd36d1c98bd3, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.532359 7815 consensus_queue.cc:260] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.532491 7814 ts_tablet_manager.cc:1428] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 11:32:57.533473 7654 heartbeater.cc:503] Master 127.6.9.254:44777 was elected leader, sending a full tablet report...
I20260929 11:32:57.534099 7815 ts_tablet_manager.cc:1428] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
W20260929 11:32:57.635780 7794 tablet.cc:2369] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:32:57.651698 7655 tablet.cc:2369] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:32:57.675843 7516 tablet.cc:2369] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:57.713727 7819 raft_consensus.cc:482] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:32:57.714041 7819 raft_consensus.cc:504] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.715692 7819 leader_election.cc:288] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:57.722481 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.716328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44813 (local address 127.6.9.195:35051)
0929 11:32:57.716520 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.716525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.717037 (+ 512us) server_negotiation.cc:407] Connection header received
0929 11:32:57.717216 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.717222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.717277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.717347 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:57.718109 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.718917 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.720815 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.721634 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.721741 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.721850 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.722121 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.722237 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:32:57.722554 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.716044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:40071 (local address 127.6.9.194:45433)
0929 11:32:57.716257 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.716264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.716999 (+ 735us) server_negotiation.cc:407] Connection header received
0929 11:32:57.717202 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.717207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.717269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.717343 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:32:57.718011 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.718805 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.720940 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.721660 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.721779 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.721880 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.722191 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.722312 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:32:57.722663 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.715912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:45433 (local address 127.6.9.193:40071)
0929 11:32:57.716884 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.716909 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.717055 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.717368 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.717372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.717394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:57.717871 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.717881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.718923 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.718928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.720792 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.720806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.721907 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.721914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.722050 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.722073 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.722146 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.722264 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":718,"threads_started":1}
I20260929 11:32:57.722684 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.715909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:35051 (local address 127.6.9.193:44813)
0929 11:32:57.716941 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.716956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.717086 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.717372 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.717376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.717436 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:32:57.717948 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.717958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.719055 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.719059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.720646 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.720682 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.721847 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.721853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.721980 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.722007 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.722086 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.722216 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":33,"spinlock_wait_cycles":31488,"thread_start_us":331,"threads_started":1}
I20260929 11:32:57.723639 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a194322979e4ed98e4d6b035c86d0c0" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3" is_pre_election: true
I20260929 11:32:57.723788 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a194322979e4ed98e4d6b035c86d0c0" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" is_pre_election: true
I20260929 11:32:57.724239 7748 raft_consensus.cc:2454] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:57.724222 7609 raft_consensus.cc:2454] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:57.724931 7396 leader_election.cc:302] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1abc8a453e0644cfa3eb33f36cf0b3af, ba87fc549df647c4a125ce0087fb24d9, e669905e605645afa81efd36d1c98bd3; no voters:
I20260929 11:32:57.725390 7819 raft_consensus.cc:2781] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:32:57.725529 7819 raft_consensus.cc:482] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:32:57.725632 7819 raft_consensus.cc:3037] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.727835 7819 raft_consensus.cc:504] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.728430 7819 leader_election.cc:288] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Requested vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:57.729028 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a194322979e4ed98e4d6b035c86d0c0" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
I20260929 11:32:57.729019 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a194322979e4ed98e4d6b035c86d0c0" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3"
I20260929 11:32:57.729221 7748 raft_consensus.cc:3037] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.729226 7609 raft_consensus.cc:3037] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.731686 7609 raft_consensus.cc:2454] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:57.731938 7748 raft_consensus.cc:2454] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:57.732194 7396 leader_election.cc:302] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba87fc549df647c4a125ce0087fb24d9, e669905e605645afa81efd36d1c98bd3; no voters:
I20260929 11:32:57.732520 7819 raft_consensus.cc:2781] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:57.732978 7819 raft_consensus.cc:686] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 1 LEADER]: Becoming Leader. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Running, Role: LEADER
I20260929 11:32:57.733464 7819 consensus_queue.cc:237] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.737826 7308 catalog_manager.cc:5495] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } health_report { overall_health: UNKNOWN } } }
I20260929 11:32:57.788695 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.783470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.6.9.193:40267)
0929 11:32:57.783704 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.783710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.783732 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:32:57.783902 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.783906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.783994 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.784128 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:57.784440 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.785193 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.786478 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.786874 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.786960 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.787493 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:57.788042 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:57.788149 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.788306 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.788371 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:32:57.799782 7748 raft_consensus.cc:1261] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:57.800805 7823 consensus_queue.cc:1035] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:57.801992 7609 raft_consensus.cc:1261] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:57.802757 7823 consensus_queue.cc:1035] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:57.814162 7826 mvcc.cc:205] Tried to move back new op lower bound from 7334631742646771712 to 7334631742399938560. Current Snapshot: MvccSnapshot[applied={T|T < 7334631742646771712}]
I20260929 11:32:57.815325 7827 mvcc.cc:205] Tried to move back new op lower bound from 7334631742646771712 to 7334631742399938560. Current Snapshot: MvccSnapshot[applied={T|T < 7334631742646771712}]
I20260929 11:32:57.835412 7308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:32:57.835714 7308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:32:57.839519 7308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Sending DeleteTablet for 3 replicas of tablet 9a194322979e4ed98e4d6b035c86d0c0
I20260929 11:32:57.840690 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:57 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:49542
I20260929 11:32:57.840692 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:57 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260929 11:32:57.841831 7723 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a194322979e4ed98e4d6b035c86d0c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:57 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47596
I20260929 11:32:57.843830 7308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:32:57.845320 7308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:32:57.855652 7444 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2e286292144f295c4a11fe6f444fd (DEFAULT_TABLE table=test-table [id=4e92ba9780a4453184d42ddfd19d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.855806 7722 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2e286292144f295c4a11fe6f444fd (DEFAULT_TABLE table=test-table [id=4e92ba9780a4453184d42ddfd19d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.856359 7722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2e286292144f295c4a11fe6f444fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.856493 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2e286292144f295c4a11fe6f444fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.860149 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 3db2e286292144f295c4a11fe6f444fd (DEFAULT_TABLE table=test-table [id=4e92ba9780a4453184d42ddfd19d9696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:32:57.860777 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db2e286292144f295c4a11fe6f444fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.863662 7812 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Bootstrap starting.
I20260929 11:32:57.863808 7814 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Bootstrap starting.
I20260929 11:32:57.864315 7836 tablet_replica.cc:330] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: stopping tablet replica
I20260929 11:32:57.865049 7836 raft_consensus.cc:2229] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:32:57.865379 7812 tablet_bootstrap.cc:654] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.865475 7814 tablet_bootstrap.cc:654] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.866084 7836 raft_consensus.cc:2258] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:57.868327 7836 ts_tablet_manager.cc:1905] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:57.871155 7837 tablet_replica.cc:330] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: stopping tablet replica
I20260929 11:32:57.871438 7815 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Bootstrap starting.
I20260929 11:32:57.874481 7838 tablet_replica.cc:330] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: stopping tablet replica
I20260929 11:32:57.875157 7838 raft_consensus.cc:2229] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:32:57.875710 7838 raft_consensus.cc:2258] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:57.875981 7815 tablet_bootstrap.cc:654] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:57.878239 7838 ts_tablet_manager.cc:1905] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:57.879988 7837 raft_consensus.cc:2229] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:32:57.881906 7837 raft_consensus.cc:2258] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:57.882336 7812 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: No bootstrap required, opened a new log
I20260929 11:32:57.882514 7812 ts_tablet_manager.cc:1397] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260929 11:32:57.883342 7812 raft_consensus.cc:348] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.883579 7812 raft_consensus.cc:374] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.883698 7812 raft_consensus.cc:729] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc8a453e0644cfa3eb33f36cf0b3af, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.884040 7812 consensus_queue.cc:260] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.885507 7812 ts_tablet_manager.cc:1428] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 11:32:57.888638 7838 ts_tablet_manager.cc:1918] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:32:57.888824 7838 log.cc:1196] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/9a194322979e4ed98e4d6b035c86d0c0
I20260929 11:32:57.889309 7838 ts_tablet_manager.cc:1939] T 9a194322979e4ed98e4d6b035c86d0c0 P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting consensus metadata
I20260929 11:32:57.889636 7814 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: No bootstrap required, opened a new log
I20260929 11:32:57.889825 7814 ts_tablet_manager.cc:1397] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.016s
I20260929 11:32:57.890349 7837 ts_tablet_manager.cc:1905] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:57.890815 7814 raft_consensus.cc:348] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.891086 7814 raft_consensus.cc:374] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.891216 7814 raft_consensus.cc:729] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.891528 7814 consensus_queue.cc:260] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.891774 7295 catalog_manager.cc:4841] TS 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051): tablet 9a194322979e4ed98e4d6b035c86d0c0 (table test-table [id=26c37479d4174d5cac9df6af7b9035bb]) successfully deleted
I20260929 11:32:57.892156 7814 ts_tablet_manager.cc:1428] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 11:32:57.893772 7815 tablet_bootstrap.cc:492] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: No bootstrap required, opened a new log
I20260929 11:32:57.893940 7815 ts_tablet_manager.cc:1397] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.003s
I20260929 11:32:57.894737 7815 raft_consensus.cc:348] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.894972 7815 raft_consensus.cc:374] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:57.895080 7815 raft_consensus.cc:729] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e669905e605645afa81efd36d1c98bd3, State: Initialized, Role: FOLLOWER
I20260929 11:32:57.895352 7815 consensus_queue.cc:260] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.895926 7815 ts_tablet_manager.cc:1428] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 11:32:57.900894 7837 ts_tablet_manager.cc:1918] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:32:57.901039 7837 log.cc:1196] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/9a194322979e4ed98e4d6b035c86d0c0
I20260929 11:32:57.901503 7837 ts_tablet_manager.cc:1939] T 9a194322979e4ed98e4d6b035c86d0c0 P e669905e605645afa81efd36d1c98bd3: Deleting consensus metadata
I20260929 11:32:57.901727 7834 raft_consensus.cc:482] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:32:57.901957 7834 raft_consensus.cc:504] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.902855 7834 leader_election.cc:288] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:57.903611 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2e286292144f295c4a11fe6f444fd" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" is_pre_election: true
I20260929 11:32:57.903882 7748 raft_consensus.cc:2454] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:57.904211 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2e286292144f295c4a11fe6f444fd" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3" is_pre_election: true
I20260929 11:32:57.904451 7609 raft_consensus.cc:2454] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:57.904876 7396 leader_election.cc:302] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba87fc549df647c4a125ce0087fb24d9, e669905e605645afa81efd36d1c98bd3; no voters:
I20260929 11:32:57.905156 7834 raft_consensus.cc:2781] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:32:57.905293 7834 raft_consensus.cc:482] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:32:57.905388 7834 raft_consensus.cc:3037] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.905794 7293 catalog_manager.cc:4841] TS e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433): tablet 9a194322979e4ed98e4d6b035c86d0c0 (table test-table [id=26c37479d4174d5cac9df6af7b9035bb]) successfully deleted
I20260929 11:32:57.908414 7834 raft_consensus.cc:504] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.909514 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2e286292144f295c4a11fe6f444fd" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3"
I20260929 11:32:57.909601 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db2e286292144f295c4a11fe6f444fd" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
I20260929 11:32:57.909711 7609 raft_consensus.cc:3037] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.909801 7748 raft_consensus.cc:3037] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:57.912583 7748 raft_consensus.cc:2454] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:57.912592 7609 raft_consensus.cc:2454] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:57.913054 7398 leader_election.cc:302] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc8a453e0644cfa3eb33f36cf0b3af, ba87fc549df647c4a125ce0087fb24d9; no voters:
I20260929 11:32:57.913370 7823 raft_consensus.cc:2781] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:57.913440 7834 leader_election.cc:288] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Requested vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:57.913573 7823 raft_consensus.cc:686] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 1 LEADER]: Becoming Leader. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Running, Role: LEADER
I20260929 11:32:57.913858 7823 consensus_queue.cc:237] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:57.916260 7836 ts_tablet_manager.cc:1918] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:32:57.916469 7836 log.cc:1196] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/9a194322979e4ed98e4d6b035c86d0c0
I20260929 11:32:57.917085 7836 ts_tablet_manager.cc:1939] T 9a194322979e4ed98e4d6b035c86d0c0 P ba87fc549df647c4a125ce0087fb24d9: Deleting consensus metadata
I20260929 11:32:57.918548 7308 catalog_manager.cc:5495] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } health_report { overall_health: UNKNOWN } } }
I20260929 11:32:57.925225 7294 catalog_manager.cc:4841] TS ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193:40267): tablet 9a194322979e4ed98e4d6b035c86d0c0 (table test-table [id=26c37479d4174d5cac9df6af7b9035bb]) successfully deleted
Sep 29 11:32:57 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681575, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:32:57.971890 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.951248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44777 (local address 127.0.0.1:47244)
0929 11:32:57.958365 (+ 7117us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:57.958388 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:57.958547 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:57.959851 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:57.959856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:57.960843 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:57.961089 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.961098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.962155 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.962160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.963115 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:57.963127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.963795 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.963801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:57.963965 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.964547 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:57.964563 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:57.966620 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:57.969478 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.969489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.969498 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.969879 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.970294 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:57.970299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:57.970302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:57.970423 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:57.970898 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:57.970906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:57.971061 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 11:32:57.971202 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 11:32:57.971453 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6933,"thread_start_us":7420,"threads_started":1}
I20260929 11:32:57.971922 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:57.951204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.6.9.254:44777)
0929 11:32:57.959359 (+ 8155us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:57.959366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:57.959390 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:32:57.959446 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:57.959451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:57.959523 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:57.959626 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:57.961242 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.962025 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.963236 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:57.963630 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:57.963729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:57.966799 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:57.966804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:57.966807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:57.966834 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:57.969363 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.970055 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.970060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.970063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.970170 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:57.970520 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:57.970525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:57.970528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:57.970785 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:57.970907 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:57.971296 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 11:32:57.971574 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":8015,"thread_start_us":9019,"threads_started":1}
I20260929 11:32:57.976893 7308 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:32:57.977934 7308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:32:57.984740 7444 tablet_service.cc:1467] Processing CreateTablet for tablet 33d773332321484da34fe6f8db14b4db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14a740b381734c9887577e709ad345dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:32:57.984717 7584 tablet_service.cc:1467] Processing CreateTablet for tablet 33d773332321484da34fe6f8db14b4db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14a740b381734c9887577e709ad345dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:32:57.985256 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d773332321484da34fe6f8db14b4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.985275 7584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d773332321484da34fe6f8db14b4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.989413 7723 tablet_service.cc:1467] Processing CreateTablet for tablet 33d773332321484da34fe6f8db14b4db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=14a740b381734c9887577e709ad345dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:32:57.989935 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d773332321484da34fe6f8db14b4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:57.992487 7815 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: Bootstrap starting.
I20260929 11:32:57.994200 7815 tablet_bootstrap.cc:654] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:58.003717 7812 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: Bootstrap starting.
I20260929 11:32:58.005673 7812 tablet_bootstrap.cc:654] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:58.012408 7814 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: Bootstrap starting.
I20260929 11:32:58.014547 7814 tablet_bootstrap.cc:654] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:58.033041 7814 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: No bootstrap required, opened a new log
I20260929 11:32:58.033254 7814 ts_tablet_manager.cc:1397] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20260929 11:32:58.035157 7814 raft_consensus.cc:348] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.035452 7814 raft_consensus.cc:374] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:58.035537 7814 raft_consensus.cc:729] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Initialized, Role: FOLLOWER
I20260929 11:32:58.035843 7814 consensus_queue.cc:260] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.036320 7823 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.036628 7823 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.039727 7814 ts_tablet_manager.cc:1428] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 11:32:58.043339 7815 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: No bootstrap required, opened a new log
I20260929 11:32:58.043597 7815 ts_tablet_manager.cc:1397] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: Time spent bootstrapping tablet: real 0.051s user 0.029s sys 0.018s
I20260929 11:32:58.044631 7812 tablet_bootstrap.cc:492] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: No bootstrap required, opened a new log
I20260929 11:32:58.044514 7815 raft_consensus.cc:348] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.044821 7812 ts_tablet_manager.cc:1397] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.008s
I20260929 11:32:58.044932 7815 raft_consensus.cc:374] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:58.045086 7815 raft_consensus.cc:729] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e669905e605645afa81efd36d1c98bd3, State: Initialized, Role: FOLLOWER
I20260929 11:32:58.045442 7815 consensus_queue.cc:260] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.045536 7812 raft_consensus.cc:348] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.045763 7812 raft_consensus.cc:374] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:58.045835 7812 raft_consensus.cc:729] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abc8a453e0644cfa3eb33f36cf0b3af, State: Initialized, Role: FOLLOWER
I20260929 11:32:58.046100 7812 consensus_queue.cc:260] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.046008 7820 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.046381 7820 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.046650 7812 ts_tablet_manager.cc:1428] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 11:32:58.047724 7815 ts_tablet_manager.cc:1428] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 11:32:58.046613 7818 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.048305 7818 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.106338 7823 raft_consensus.cc:482] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:32:58.106564 7823 raft_consensus.cc:504] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.107244 7823 leader_election.cc:288] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:58.107813 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d773332321484da34fe6f8db14b4db" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3" is_pre_election: true
I20260929 11:32:58.107962 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d773332321484da34fe6f8db14b4db" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" is_pre_election: true
I20260929 11:32:58.108100 7609 raft_consensus.cc:2454] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:58.108218 7748 raft_consensus.cc:2454] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 0.
I20260929 11:32:58.108721 7396 leader_election.cc:302] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1abc8a453e0644cfa3eb33f36cf0b3af, ba87fc549df647c4a125ce0087fb24d9, e669905e605645afa81efd36d1c98bd3; no voters:
I20260929 11:32:58.109040 7823 raft_consensus.cc:2781] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:32:58.109148 7823 raft_consensus.cc:482] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:32:58.109263 7823 raft_consensus.cc:3037] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:58.111788 7823 raft_consensus.cc:504] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.113003 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d773332321484da34fe6f8db14b4db" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af"
I20260929 11:32:58.112989 7609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d773332321484da34fe6f8db14b4db" candidate_uuid: "ba87fc549df647c4a125ce0087fb24d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e669905e605645afa81efd36d1c98bd3"
I20260929 11:32:58.113247 7748 raft_consensus.cc:3037] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:58.113255 7609 raft_consensus.cc:3037] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:58.116142 7748 raft_consensus.cc:2454] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:58.116384 7609 raft_consensus.cc:2454] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba87fc549df647c4a125ce0087fb24d9 in term 1.
I20260929 11:32:58.116732 7398 leader_election.cc:302] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abc8a453e0644cfa3eb33f36cf0b3af, ba87fc549df647c4a125ce0087fb24d9; no voters:
I20260929 11:32:58.117053 7834 raft_consensus.cc:2781] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:58.117185 7834 raft_consensus.cc:686] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [term 1 LEADER]: Becoming Leader. State: Replica: ba87fc549df647c4a125ce0087fb24d9, State: Running, Role: LEADER
I20260929 11:32:58.117455 7834 consensus_queue.cc:237] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } }
I20260929 11:32:58.119014 7819 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: TxnStatusTablet state changed. Reason: New leader ba87fc549df647c4a125ce0087fb24d9. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.119313 7819 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:32:58.121109 7823 leader_election.cc:288] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [CANDIDATE]: Term 1 election: Requested vote from peers e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433), 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051)
I20260929 11:32:58.121264 7308 catalog_manager.cc:5495] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } health_report { overall_health: UNKNOWN } } }
I20260929 11:32:58.132467 7854 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:32:58.132790 7854 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:32:58.142937 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.137650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.6.9.193:40267)
0929 11:32:58.137831 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.137838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.138012 (+ 174us) server_negotiation.cc:407] Connection header received
0929 11:32:58.138249 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.138255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.138326 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.138403 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:58.138770 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.139597 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.140965 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.141626 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.141748 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.142204 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:58.142337 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:58.142419 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.142563 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.142644 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:32:58.142969 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.137603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:40267 (local address 127.0.0.1:54062)
0929 11:32:58.137920 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:58.137936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:58.138079 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:58.138448 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:58.138453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:58.138464 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:32:58.138638 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:58.138646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.139721 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.139727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:58.140785 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:58.140801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.141811 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.141818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:58.141977 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.142032 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:32:58.142433 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:32:58.142454 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 11:32:58.142529 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:32:58.142679 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 11:32:58.146713 7609 raft_consensus.cc:1261] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:58.146919 7748 raft_consensus.cc:1261] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:58.147505 7819 consensus_queue.cc:1035] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:58.147775 7823 consensus_queue.cc:1035] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:58.159291 7820 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: TxnStatusTablet state changed. Reason: New leader ba87fc549df647c4a125ce0087fb24d9. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.159382 7818 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: TxnStatusTablet state changed. Reason: New leader ba87fc549df647c4a125ce0087fb24d9. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.159607 7818 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.159611 7820 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.162504 7819 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.162770 7819 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:32:58.167203 7818 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.167452 7818 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P 1abc8a453e0644cfa3eb33f36cf0b3af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.174607 7823 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.174889 7823 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P ba87fc549df647c4a125ce0087fb24d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:32:58.179149 7820 tablet_replica.cc:439] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba87fc549df647c4a125ce0087fb24d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba87fc549df647c4a125ce0087fb24d9" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 40267 } } peers { permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 } } peers { permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 } } }
I20260929 11:32:58.179426 7820 tablet_replica.cc:442] T 33d773332321484da34fe6f8db14b4db P e669905e605645afa81efd36d1c98bd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:32:58.193519 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.188047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:50927 (local address 127.6.9.193:40267)
0929 11:32:58.188420 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.188426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.188821 (+ 395us) server_negotiation.cc:407] Connection header received
0929 11:32:58.188999 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.189004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.189071 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.189134 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:58.189630 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.190369 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.191824 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.192259 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.192358 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.192787 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:58.192890 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:58.192959 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.193072 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.193144 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":235}
I20260929 11:32:58.193822 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.188043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:40267 (local address 127.6.9.193:50927)
0929 11:32:58.188720 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:58.188747 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:58.188885 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:58.189159 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:58.189164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:58.189174 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:32:58.189504 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:58.189513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.190476 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.190482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:58.191685 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:58.191700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.192423 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.192430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:58.192555 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.192604 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:32:58.192971 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:32:58.192981 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:32:58.193052 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 11:32:58.193232 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":388,"threads_started":1}
I20260929 11:32:58.205549 7609 raft_consensus.cc:1261] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:58.205605 7747 raft_consensus.cc:1261] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Refusing update from remote peer ba87fc549df647c4a125ce0087fb24d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:58.206161 7819 consensus_queue.cc:1035] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e669905e605645afa81efd36d1c98bd3" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:58.206737 7819 consensus_queue.cc:1035] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abc8a453e0644cfa3eb33f36cf0b3af" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:58.254058 7828 mvcc.cc:205] Tried to move back new op lower bound from 7334631744324694016 to 7334631743138103296. Current Snapshot: MvccSnapshot[applied={T|T < 7334631744324694016}]
I20260929 11:32:58.290977 7307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:32:58.291222 7307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:32:58.294070 7307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32f6e4e5ec3141eda43ba26ed1a2293c: Sending DeleteTablet for 3 replicas of tablet 3db2e286292144f295c4a11fe6f444fd
I20260929 11:32:58.294759 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db2e286292144f295c4a11fe6f444fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:58 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260929 11:32:58.295049 7723 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db2e286292144f295c4a11fe6f444fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:58 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47596
I20260929 11:32:58.295102 7836 tablet_replica.cc:330] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: stopping tablet replica
I20260929 11:32:58.295318 7836 raft_consensus.cc:2229] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:32:58.295531 7584 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db2e286292144f295c4a11fe6f444fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:32:58 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:49542
I20260929 11:32:58.295642 7836 raft_consensus.cc:2258] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:58.295826 7837 tablet_replica.cc:330] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: stopping tablet replica
I20260929 11:32:58.295955 7838 tablet_replica.cc:330] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: stopping tablet replica
I20260929 11:32:58.296011 7837 raft_consensus.cc:2229] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:32:58.296151 7838 raft_consensus.cc:2229] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:32:58.296197 7837 raft_consensus.cc:2258] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:58.296330 7838 raft_consensus.cc:2258] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:32:58.297565 7837 ts_tablet_manager.cc:1905] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:58.297627 7838 ts_tablet_manager.cc:1905] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:58.305060 7837 ts_tablet_manager.cc:1918] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:32:58.305238 7837 log.cc:1196] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/3db2e286292144f295c4a11fe6f444fd
I20260929 11:32:58.305233 7836 ts_tablet_manager.cc:1905] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:32:58.305666 7837 ts_tablet_manager.cc:1939] T 3db2e286292144f295c4a11fe6f444fd P e669905e605645afa81efd36d1c98bd3: Deleting consensus metadata
I20260929 11:32:58.307018 7293 catalog_manager.cc:4841] TS e669905e605645afa81efd36d1c98bd3 (127.6.9.194:45433): tablet 3db2e286292144f295c4a11fe6f444fd (table test-table [id=4e92ba9780a4453184d42ddfd19d9696]) successfully deleted
I20260929 11:32:58.312646 7836 ts_tablet_manager.cc:1918] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:32:58.312983 7836 log.cc:1196] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/3db2e286292144f295c4a11fe6f444fd
I20260929 11:32:58.313504 7836 ts_tablet_manager.cc:1939] T 3db2e286292144f295c4a11fe6f444fd P ba87fc549df647c4a125ce0087fb24d9: Deleting consensus metadata
I20260929 11:32:58.313480 7838 ts_tablet_manager.cc:1918] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:32:58.313643 7838 log.cc:1196] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/3db2e286292144f295c4a11fe6f444fd
I20260929 11:32:58.314061 7838 ts_tablet_manager.cc:1939] T 3db2e286292144f295c4a11fe6f444fd P 1abc8a453e0644cfa3eb33f36cf0b3af: Deleting consensus metadata
I20260929 11:32:58.314836 7294 catalog_manager.cc:4841] TS ba87fc549df647c4a125ce0087fb24d9 (127.6.9.193:40267): tablet 3db2e286292144f295c4a11fe6f444fd (table test-table [id=4e92ba9780a4453184d42ddfd19d9696]) successfully deleted
I20260929 11:32:58.315220 7295 catalog_manager.cc:4841] TS 1abc8a453e0644cfa3eb33f36cf0b3af (127.6.9.195:35051): tablet 3db2e286292144f295c4a11fe6f444fd (table test-table [id=4e92ba9780a4453184d42ddfd19d9696]) successfully deleted
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:32:58.324223 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.308119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.6.9.193:40267)
0929 11:32:58.308326 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.308334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.308846 (+ 512us) server_negotiation.cc:407] Connection header received
0929 11:32:58.309105 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.309109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.309173 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.309254 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:58.310605 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.311386 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.312603 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.313312 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.313438 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.318330 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:58.318342 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:58.318358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:58.318404 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:58.321448 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.322082 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.322090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.322093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.322232 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.322581 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.322584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.322586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.323042 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:58.323347 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.323665 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.323885 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 11:32:58.324915 7510 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54076
I20260929 11:32:58.339237 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.327562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.6.9.254:44777)
0929 11:32:58.327765 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.327770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.327960 (+ 190us) server_negotiation.cc:407] Connection header received
0929 11:32:58.328213 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.328217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.328297 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.328384 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:58.329693 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.330443 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.331588 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.332021 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.332133 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.334244 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:58.334249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:58.334252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:58.334282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:58.336847 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.337448 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.337453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.337456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.337579 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.337916 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.337921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.337925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.338200 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:58.338437 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.338712 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.338934 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 11:32:58.339751 7307 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47246
I20260929 11:32:58.341348 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7380
I20260929 11:32:58.355891 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.342406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.6.9.193:40267)
0929 11:32:58.342814 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.342820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.342839 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:32:58.342900 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.342905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.342980 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.343101 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:58.344101 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.345280 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.346417 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.347112 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.347241 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.349562 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:58.349570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:58.349575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:58.349614 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:58.353147 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.353821 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.353826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.353830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.353994 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.354377 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.354382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.354385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.354783 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:58.355053 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.355317 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.355543 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
W20260929 11:32:58.356525 7510 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54080
W20260929 11:32:58.357074 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:58.357239 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7380
I20260929 11:32:58.373868 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7519
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:32:58.389991 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.374672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.6.9.194:45433)
0929 11:32:58.375207 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.375212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.375230 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:32:58.375291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.375294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.375366 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.375478 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:58.376755 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.377538 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.378876 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.379338 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.379443 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.383641 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:58.383660 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:58.383675 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:58.383716 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:58.386924 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.387803 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.387813 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.387816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.387963 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.388350 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.388354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.388356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.388843 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:58.389141 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.389434 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.389667 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
W20260929 11:32:58.390590 7649 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49548
W20260929 11:32:58.391192 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7519: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:58.391352 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7519
I20260929 11:32:58.410967 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7658
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:32:58.426784 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:58.411811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.6.9.195:35051)
0929 11:32:58.412559 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:58.412564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:58.412580 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:32:58.412638 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:58.412642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:58.412730 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:58.412829 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:58.414176 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.415025 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.416293 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:58.416934 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:58.417053 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:58.421139 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:58.421155 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:58.421170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:58.421215 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:58.423941 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.424740 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.424750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.424754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.424897 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:58.425261 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:58.425267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:58.425270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:58.425634 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:58.425929 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:58.426210 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 11:32:58.426432 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":493,"threads_started":1}
W20260929 11:32:58.427454 7788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47612
W20260929 11:32:58.428066 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:58.428198 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7658
I20260929 11:32:58.441336 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7277
I20260929 11:32:58.442519 7349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:58.554593 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7277
2026-09-29T11:32:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4263 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7874](info): set up 2 sockets
Sep 29 11:32:58 dist-test-slave-c0l9 krb5kdc[7874](info): commencing operation
krb5kdc: starting...
W20260929 11:32:59.544477 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:32:59 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:32:59.713963 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:34919
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:00.034119 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:00.034466 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:00.034611 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:00.042029 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:00.042182 7887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:33:00.042256 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:00.042341 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:00.042408 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:00.042493 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:00.053001 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:34919
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:00.054287 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:00.055588 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:00.067098 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:00.067109 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:00.067452 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:00.067988 7887 server_base.cc:1034] running on GCE node
I20260929 11:33:00.068881 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:00.070348 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:00.071563 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790681580071495 us; error 41 us; skew 500 ppm
I20260929 11:33:00.071905 7887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:00 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:00.075054 7887 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:00.076606 7887 webserver.cc:460] Webserver started at http://127.6.9.254:37357/ using document root <none> and password file <none>
I20260929 11:33:00.077363 7887 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:00.077468 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:00.077808 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:00.081079 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "8ffe1ae3fa954694a958bea6deb52ec5"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "b223cbcdb0889b0cdb5774a19ea35d9d"
server_key_iv: "d7bd9cf49df7ab61785984dc00f22c92"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.081908 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "8ffe1ae3fa954694a958bea6deb52ec5"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "b223cbcdb0889b0cdb5774a19ea35d9d"
server_key_iv: "d7bd9cf49df7ab61785984dc00f22c92"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.087414 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:33:00.091151 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:00.092542 7887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:00.092742 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "8ffe1ae3fa954694a958bea6deb52ec5"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "b223cbcdb0889b0cdb5774a19ea35d9d"
server_key_iv: "d7bd9cf49df7ab61785984dc00f22c92"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.092958 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:00.126394 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:00.130710 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:00.130964 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:00.149588 7887 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:34919
I20260929 11:33:00.149606 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:34919 every 8 connection(s)
I20260929 11:33:00.150918 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:00.154672 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:00.160202 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7887
I20260929 11:33:00.160394 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:00.161053 6183 external_mini_cluster.cc:1428] Setting key 9809e1e79aa2b126f17d5e8bb48977b7
I20260929 11:33:00.165587 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Bootstrap starting.
I20260929 11:33:00.169459 7964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:00.170892 7964 log.cc:824] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:00.175022 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: No bootstrap required, opened a new log
Sep 29 11:33:00 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:00.183002 7964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } }
I20260929 11:33:00.183409 7964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:00.183506 7964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffe1ae3fa954694a958bea6deb52ec5, State: Initialized, Role: FOLLOWER
I20260929 11:33:00.184226 7964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } }
I20260929 11:33:00.184512 7964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:00.184654 7964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:00.184859 7964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:00.185761 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:00.163031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.6.9.254:34919)
0929 11:33:00.163847 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:00.163858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:00.163901 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:33:00.165651 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:00.165676 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:00.166142 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:00.166484 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:00.168406 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.169876 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.171861 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.173011 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.173222 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:00.177643 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:00.177654 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:00.177673 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:00.177716 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:00.180852 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.181631 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.181639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.181652 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.181840 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.182386 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.182390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.182392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.182992 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:00.183303 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:00.183823 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 11:33:00.184273 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":370,"threads_started":1}
I20260929 11:33:00.187999 7964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } }
I20260929 11:33:00.188530 7964 leader_election.cc:302] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ffe1ae3fa954694a958bea6deb52ec5; no voters:
I20260929 11:33:00.189214 7964 leader_election.cc:288] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:00.189343 7969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:00.190222 7969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 8ffe1ae3fa954694a958bea6deb52ec5, State: Running, Role: LEADER
I20260929 11:33:00.190728 7969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } }
I20260929 11:33:00.191093 7964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:00.193186 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ffe1ae3fa954694a958bea6deb52ec5. Latest consensus state: current_term: 1 leader_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } } }
I20260929 11:33:00.193523 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:00.194025 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffe1ae3fa954694a958bea6deb52ec5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 34919 } } }
I20260929 11:33:00.194437 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:00.194814 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:00.203995 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:00.213270 7974 catalog_manager.cc:1347] Generated new cluster ID: 5a2b410e5a254578bb63c74beb3c9dd0
I20260929 11:33:00.213397 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:00.235039 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:00.236377 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:00.243561 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Generated new TSK 0
I20260929 11:33:00.244630 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:00.290406 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:34919
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:00.611656 7990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:00.611943 7990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:00.612048 7990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:00.619338 7990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:00.619463 7990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:00.619549 7990 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:33:00.619599 7990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:00.619670 7990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:00.629475 7990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:00.630723 7990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:00.631999 7990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:00.644906 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:00.644904 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:00.644934 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:00.645427 7990 server_base.cc:1034] running on GCE node
I20260929 11:33:00.646245 7990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:00.647260 7990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:00.648485 7990 hybrid_clock.cc:648] HybridClock initialized: now 1790681580648421 us; error 54 us; skew 500 ppm
I20260929 11:33:00.648846 7990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:00 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:00.652182 7990 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:00.653769 7990 webserver.cc:460] Webserver started at http://127.6.9.193:39431/ using document root <none> and password file <none>
I20260929 11:33:00.654479 7990 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:00.654587 7990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:00.654904 7990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:00.657940 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "0cceb281db874057995c3ea5a56e7127"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "f1c591a8dc7faf3f4e3b157ca48711eb"
server_key_iv: "edcb95c9162e348db15de56d7e20a1a1"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.658695 7990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "0cceb281db874057995c3ea5a56e7127"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "f1c591a8dc7faf3f4e3b157ca48711eb"
server_key_iv: "edcb95c9162e348db15de56d7e20a1a1"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.664214 7990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:33:00.668047 8005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:00.669517 7990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:00.669708 7990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "0cceb281db874057995c3ea5a56e7127"
format_stamp: "Formatted at 2026-09-29 11:33:00 on dist-test-slave-c0l9"
server_key: "f1c591a8dc7faf3f4e3b157ca48711eb"
server_key_iv: "edcb95c9162e348db15de56d7e20a1a1"
server_key_version: "encryptionkey@0"
I20260929 11:33:00.669900 7990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:00.682773 7990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:00.686964 7990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:00.687244 7990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:00.689987 7990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:00.690166 7990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:00.690297 7990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:00.690376 7990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:33:00 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:00.713316 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:00.690419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:36731 (local address 127.6.9.254:34919)
0929 11:33:00.690979 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:00.690986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:00.692046 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 11:33:00.694194 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:00.694200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:00.694283 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:00.694379 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:00.696684 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.697679 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.700088 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.700536 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.700706 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:00.705851 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:00.705856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:00.705860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:00.705897 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:00.708991 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.710085 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.710103 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.710106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.710267 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.710887 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.710893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.710896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.711261 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:00.711426 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:00.712611 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0929 11:33:00.712906 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":252,"threads_started":1}
I20260929 11:33:00.714299 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:00.691004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.193:36731)
0929 11:33:00.691873 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:00.691926 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:00.693879 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:00.694682 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:00.694695 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:00.695522 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:00.696479 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:00.696505 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.697866 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.697870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:00.699944 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:00.699960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.700835 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.700843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:00.701240 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:00.702247 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:00.702280 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:00.705599 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:00.709148 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:00.709162 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:00.709185 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:00.709894 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:00.710420 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:00.710430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:00.710435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:00.710763 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:00.711516 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:00.711525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:00.712071 (+ 546us) client_negotiation.cc:769] Sending connection context
0929 11:33:00.712515 (+ 444us) client_negotiation.cc:240] Negotiation successful
0929 11:33:00.713028 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":362,"threads_started":1}
I20260929 11:33:00.734277 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:00.719677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.193:34775)
0929 11:33:00.721236 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:00.721257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:00.721413 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:00.721953 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:00.721958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:00.722587 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:00.723034 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:00.723051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.724238 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.724246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:00.725852 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:00.725867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.726593 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.726600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:00.726771 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:00.727435 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:00.727456 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:00.728167 (+ 711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:00.731408 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:00.731416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:00.731421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:00.731959 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:00.732391 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:00.732396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:00.732400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:00.732574 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:00.733205 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:00.733210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:00.733438 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 11:33:00.733610 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 11:33:00.733878 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":1205,"threads_started":1}
I20260929 11:33:00.735059 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:00.719771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:34775 (local address 127.6.9.254:34919)
0929 11:33:00.719971 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:00.719977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:00.721352 (+ 1375us) server_negotiation.cc:407] Connection header received
0929 11:33:00.721609 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:00.721613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:00.721684 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:00.721775 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:00.723200 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.724095 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.725982 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:00.726414 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:00.726524 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:00.728331 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:00.728336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:00.728340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:00.728370 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:00.731289 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.732091 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.732097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.732100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.732259 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:00.732705 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:00.732711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:00.732714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:00.733077 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:00.733217 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:00.734599 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0929 11:33:00.734831 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:33:00.737659 7990 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:35241
I20260929 11:33:00.737735 8125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:35241 every 8 connection(s)
I20260929 11:33:00.738862 7990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:00.740636 8126 heartbeater.cc:347] Connected to a master server at 127.6.9.254:34919
I20260929 11:33:00.740863 8126 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:00.741428 8126 heartbeater.cc:511] Master 127.6.9.254:34919 requested a full tablet report, sending...
I20260929 11:33:00.743301 7919 ts_manager.cc:194] Registered new tserver with Master: 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241)
I20260929 11:33:00.744837 7919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:36731
I20260929 11:33:00.748373 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 7990
I20260929 11:33:00.748540 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:00.749174 6183 external_mini_cluster.cc:1428] Setting key dbefbb82f655851564113f568ead3bc1
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:00.785660 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:34919
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:01.098209 8130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:01.098601 8130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:01.098780 8130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:01.106314 8130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:01.106428 8130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:01.106520 8130 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:33:01.106570 8130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:01.106639 8130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:01.116760 8130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:01.118080 8130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:01.119393 8130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:01.131858 8135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:01.131822 8136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:01.131960 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:01.132944 8130 server_base.cc:1034] running on GCE node
I20260929 11:33:01.133565 8130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:01.134469 8130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:01.135677 8130 hybrid_clock.cc:648] HybridClock initialized: now 1790681581135618 us; error 51 us; skew 500 ppm
I20260929 11:33:01.135984 8130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:01.139091 8130 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:01.140527 8130 webserver.cc:460] Webserver started at http://127.6.9.194:38379/ using document root <none> and password file <none>
I20260929 11:33:01.141264 8130 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:01.141383 8130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:01.141675 8130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:01.144608 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "4c6f8dcff5f445b8910fa552dee5e178"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "08de359a9ac7ad37b99e02b4e09b3333"
server_key_iv: "22325c78458e06097ccb73b9cd8e78fb"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.145465 8130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6f8dcff5f445b8910fa552dee5e178"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "08de359a9ac7ad37b99e02b4e09b3333"
server_key_iv: "22325c78458e06097ccb73b9cd8e78fb"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.151481 8130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:33:01.155778 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:01.157372 8130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:01.157555 8130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "4c6f8dcff5f445b8910fa552dee5e178"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "08de359a9ac7ad37b99e02b4e09b3333"
server_key_iv: "22325c78458e06097ccb73b9cd8e78fb"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.157780 8130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:01.183785 8130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:01.188024 8130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:01.188282 8130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:01.190943 8130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:01.191057 8130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:01.191165 8130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:01.191237 8130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:01.214409 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.191579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:39459 (local address 127.6.9.254:34919)
0929 11:33:01.191782 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.191788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.193247 (+ 1459us) server_negotiation.cc:407] Connection header received
0929 11:33:01.195226 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.195232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.195349 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.195434 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:01.197614 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.198791 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.201114 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.201684 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.201811 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.206691 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:01.206696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:01.206700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:01.206733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:01.210278 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.211344 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.211350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.211353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.211533 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.212134 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.212138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.212140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.212523 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:01.212719 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.213779 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.214027 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:33:01.216207 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.192183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.194:39459)
0929 11:33:01.193063 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.193116 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.194943 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.195732 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.195747 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.196500 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:01.197387 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.197415 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.198984 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.198989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.200967 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.200980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.202168 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.202179 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.202468 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.203350 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:01.203392 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:01.206451 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:01.210461 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.210472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.210487 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.211166 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.211646 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.211651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.211655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.211982 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.212690 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:01.212700 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:01.213278 (+ 578us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.213647 (+ 369us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.214293 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":349,"threads_started":1}
I20260929 11:33:01.235728 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.221588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:46907 (local address 127.6.9.254:34919)
0929 11:33:01.221758 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.221764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.222479 (+ 715us) server_negotiation.cc:407] Connection header received
0929 11:33:01.222647 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.222653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.222730 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.222806 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:01.223689 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.224424 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.226231 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.226898 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.227033 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.229314 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:01.229321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:01.229327 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:01.229368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:01.232830 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.233735 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.233740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.233743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.233883 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.234229 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.234233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.234236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.234658 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:01.234803 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.235188 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.235393 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:33:01.235839 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.221639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.194:46907)
0929 11:33:01.222378 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.222401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.222534 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.222842 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.222847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.223286 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:01.223561 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.223569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.224557 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.224563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.226026 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.226038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.227073 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.227078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.227201 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.227838 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:01.227865 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:01.229060 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:01.233003 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.233009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.233015 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.233628 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.233985 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.233990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.233993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.234138 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.234775 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:01.234780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:01.234950 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.235072 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.235416 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":385,"threads_started":1}
I20260929 11:33:01.237675 8130 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:40891
I20260929 11:33:01.237722 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:40891 every 8 connection(s)
I20260929 11:33:01.239078 8130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:01.241046 8265 heartbeater.cc:347] Connected to a master server at 127.6.9.254:34919
I20260929 11:33:01.241245 8265 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:01.241835 8265 heartbeater.cc:511] Master 127.6.9.254:34919 requested a full tablet report, sending...
I20260929 11:33:01.243027 7919 ts_manager.cc:194] Registered new tserver with Master: 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194:40891)
I20260929 11:33:01.243044 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8130
I20260929 11:33:01.243178 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:01.243598 6183 external_mini_cluster.cc:1428] Setting key 22f41fb0b0ed871d93b4289ecab11919
I20260929 11:33:01.243882 7919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:39459
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:01.280529 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:34919
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:01.591091 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:01.591409 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:01.591523 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:01.598901 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:01.599021 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:01.599152 8269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:33:01.599215 8269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:01.599298 8269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:01.609225 8269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:01.610620 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:01.611919 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:01.624770 8275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:01.624841 8277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:01.624775 8274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:01.625816 8269 server_base.cc:1034] running on GCE node
I20260929 11:33:01.626391 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:01.627259 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:01.628458 8269 hybrid_clock.cc:648] HybridClock initialized: now 1790681581628414 us; error 34 us; skew 500 ppm
I20260929 11:33:01.628783 8269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:01.631668 8269 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:01.633158 8269 webserver.cc:460] Webserver started at http://127.6.9.195:33039/ using document root <none> and password file <none>
I20260929 11:33:01.633829 8269 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:01.633951 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:01.634258 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:01.637140 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "7aa7571192eb43a6b5512590d223a8b9"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "d4c2f7b9ea035d236b0b715eb46e80e2"
server_key_iv: "1cb3aab5ae8bc34a82c1cafe22ffaa6a"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.637853 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "7aa7571192eb43a6b5512590d223a8b9"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "d4c2f7b9ea035d236b0b715eb46e80e2"
server_key_iv: "1cb3aab5ae8bc34a82c1cafe22ffaa6a"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.642827 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:33:01.646291 8284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:01.647575 8269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:33:01.647725 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "7aa7571192eb43a6b5512590d223a8b9"
format_stamp: "Formatted at 2026-09-29 11:33:01 on dist-test-slave-c0l9"
server_key: "d4c2f7b9ea035d236b0b715eb46e80e2"
server_key_iv: "1cb3aab5ae8bc34a82c1cafe22ffaa6a"
server_key_version: "encryptionkey@0"
I20260929 11:33:01.647915 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:01.677423 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:01.681551 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:01.681828 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:01.684489 8269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:01.684593 8269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:01.684728 8269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:01.684815 8269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:01.707069 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.685048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:48355 (local address 127.6.9.254:34919)
0929 11:33:01.685253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.685260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.686645 (+ 1385us) server_negotiation.cc:407] Connection header received
0929 11:33:01.688755 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.688761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.688859 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.688980 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:01.690476 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.691662 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.694229 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.694904 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.695042 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.699914 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:01.699920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:01.699923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:01.699968 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:01.703187 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.704030 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.704036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.704040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.704189 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.704648 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.704653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.704657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.704998 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:01.705162 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.706359 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.706674 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:33:01.708808 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.685581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.195:48355)
0929 11:33:01.686446 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.686497 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.688421 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.689224 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.689232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.689741 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:01.690313 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.690332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.691849 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.691855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.694049 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.694064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.695169 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.695177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.695491 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.696322 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:01.696354 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:01.699610 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:01.703366 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.703376 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.703394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.703891 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.704307 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.704312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.704316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.704532 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.705149 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:01.705160 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:01.705747 (+ 587us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.706176 (+ 429us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.706912 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":368,"threads_started":1}
I20260929 11:33:01.729852 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.715180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:43765 (local address 127.6.9.254:34919)
0929 11:33:01.715357 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.715363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.715532 (+ 169us) server_negotiation.cc:407] Connection header received
0929 11:33:01.715735 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.715740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.715820 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.715893 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:01.717193 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.718005 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.720100 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.720587 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.720826 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.723579 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:01.723585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:01.723588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:01.723628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:01.726690 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.727706 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.727712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.727717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.727906 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.728283 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.728288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.728292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.728687 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:01.728840 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.729425 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.729597 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:33:01.730114 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.714673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.6.9.195:43765)
0929 11:33:01.715418 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.715442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.715612 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.715943 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.715948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.716569 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:01.717006 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.717022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.718144 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.718150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.719913 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.719932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.720913 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.720919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.721121 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.721942 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:01.722022 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:01.723362 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:01.726859 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.726864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.726872 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.727518 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.728034 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:01.728038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:01.728040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:01.728188 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:01.728901 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:01.728905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:01.729151 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.729304 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.729636 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":457,"threads_started":1}
I20260929 11:33:01.740464 8269 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:46271
I20260929 11:33:01.740491 8403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:46271 every 8 connection(s)
I20260929 11:33:01.741721 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:01.743763 8404 heartbeater.cc:347] Connected to a master server at 127.6.9.254:34919
I20260929 11:33:01.743981 8404 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:01.744619 8404 heartbeater.cc:511] Master 127.6.9.254:34919 requested a full tablet report, sending...
I20260929 11:33:01.745965 7919 ts_manager.cc:194] Registered new tserver with Master: 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:01.746723 7919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:48355
I20260929 11:33:01.747447 8126 heartbeater.cc:503] Master 127.6.9.254:34919 was elected leader, sending a full tablet report...
I20260929 11:33:01.748256 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8269
I20260929 11:33:01.748402 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:01.748907 6183 external_mini_cluster.cc:1428] Setting key fee8dd93c029770941215b749e44aac8
I20260929 11:33:01.751583 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:33:01 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:01.795742 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.781719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.6.9.254:34919)
0929 11:33:01.781991 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.781998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.782277 (+ 279us) server_negotiation.cc:407] Connection header received
0929 11:33:01.782467 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.782472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.782539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.782623 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:01.784024 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.784815 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.786053 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.786487 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.786603 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.790419 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:01.790425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:01.790428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:01.790468 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:01.793148 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.793792 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.793797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.793799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.793925 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:01.794298 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:01.794302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:01.794304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:01.794597 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:01.794820 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.795136 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.795382 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 11:33:01.799980 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:01.806913 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:01.834827 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.819830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.6.9.194:40891)
0929 11:33:01.820380 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.820390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.825674 (+ 5284us) server_negotiation.cc:407] Connection header received
0929 11:33:01.826084 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.826094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.826368 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.826529 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:01.828876 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.830102 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.832820 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.833743 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.833864 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.834085 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.834380 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.834544 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
I20260929 11:33:01.835196 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.819283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40891 (local address 127.0.0.1:47826)
0929 11:33:01.825516 (+ 6233us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.825563 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.825833 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.826850 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.826856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.826898 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:01.827976 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.827987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.830244 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.830250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.832643 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.832653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.833938 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.833945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.834135 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.834195 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.834299 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.834511 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6053,"mutex_wait_us":868,"spinlock_wait_cycles":12928,"thread_start_us":5771,"threads_started":1}
I20260929 11:33:01.835304 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.819298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.6.9.193:35241)
0929 11:33:01.825003 (+ 5705us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.825014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.825665 (+ 651us) server_negotiation.cc:407] Connection header received
0929 11:33:01.826088 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.826095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.826367 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.826534 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:01.828162 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.829652 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.832551 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.834017 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.834128 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.834317 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.834865 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.835027 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5508,"thread_start_us":5372,"threads_started":1}
I20260929 11:33:01.836037 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.819144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:35241 (local address 127.0.0.1:34684)
0929 11:33:01.825517 (+ 6373us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.825560 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.825826 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.826838 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.826846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.826895 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:01.827357 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.827370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.829835 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.829844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.832388 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.832403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.834467 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.834473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.834651 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.834682 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.835502 (+ 820us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.835639 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6098,"mutex_wait_us":51,"spinlock_wait_cycles":21760,"thread_start_us":6048,"threads_started":1}
I20260929 11:33:01.836618 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.825983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:46271 (local address 127.0.0.1:60976)
0929 11:33:01.828959 (+ 2976us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.828978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.829095 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.829665 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.829670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.829694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:01.830162 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.830178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.831648 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.831653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.833805 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.833820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.835860 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.835866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.836051 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.836082 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.836163 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.836292 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2861,"mutex_wait_us":54,"thread_start_us":2727,"threads_started":1}
I20260929 11:33:01.837359 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.826348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.6.9.195:46271)
0929 11:33:01.828688 (+ 2340us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.828698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.829090 (+ 392us) server_negotiation.cc:407] Connection header received
0929 11:33:01.829223 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.829231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.829476 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.829671 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:01.830379 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.831476 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.834019 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.835406 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.835520 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.835720 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.836903 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.837055 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2159,"thread_start_us":1983,"threads_started":1}
I20260929 11:33:01.838991 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 14afc8cc25774392a525a40cac75ccc1 (DEFAULT_TABLE table=test-table [id=273c28247a4747a3a1d1074945ad7e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:01.839229 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 14afc8cc25774392a525a40cac75ccc1 (DEFAULT_TABLE table=test-table [id=273c28247a4747a3a1d1074945ad7e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:01.840080 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14afc8cc25774392a525a40cac75ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:01.840500 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14afc8cc25774392a525a40cac75ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:01.840437 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 14afc8cc25774392a525a40cac75ccc1 (DEFAULT_TABLE table=test-table [id=273c28247a4747a3a1d1074945ad7e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:01.841785 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14afc8cc25774392a525a40cac75ccc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:01.851883 8423 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Bootstrap starting.
I20260929 11:33:01.854964 8423 tablet_bootstrap.cc:654] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:01.855552 8425 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Bootstrap starting.
I20260929 11:33:01.856449 8423 log.cc:824] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:01.857772 8424 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Bootstrap starting.
I20260929 11:33:01.859046 8425 tablet_bootstrap.cc:654] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:01.860255 8425 log.cc:824] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:01.861194 8424 tablet_bootstrap.cc:654] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:01.862350 8424 log.cc:824] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:01.862540 8423 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: No bootstrap required, opened a new log
I20260929 11:33:01.862840 8423 ts_tablet_manager.cc:1397] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260929 11:33:01.863337 8425 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: No bootstrap required, opened a new log
I20260929 11:33:01.863621 8425 ts_tablet_manager.cc:1397] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 11:33:01.870759 8423 raft_consensus.cc:348] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.871295 8423 raft_consensus.cc:374] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:01.871402 8423 raft_consensus.cc:729] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Initialized, Role: FOLLOWER
I20260929 11:33:01.872131 8423 consensus_queue.cc:260] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.873226 8265 heartbeater.cc:503] Master 127.6.9.254:34919 was elected leader, sending a full tablet report...
I20260929 11:33:01.873895 8423 ts_tablet_manager.cc:1428] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Time spent starting tablet: real 0.011s user 0.010s sys 0.003s
I20260929 11:33:01.874189 8425 raft_consensus.cc:348] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.874601 8425 raft_consensus.cc:374] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:01.874707 8425 raft_consensus.cc:729] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa7571192eb43a6b5512590d223a8b9, State: Initialized, Role: FOLLOWER
I20260929 11:33:01.874796 8424 tablet_bootstrap.cc:492] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: No bootstrap required, opened a new log
I20260929 11:33:01.875105 8424 ts_tablet_manager.cc:1397] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260929 11:33:01.875306 8425 consensus_queue.cc:260] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.876396 8404 heartbeater.cc:503] Master 127.6.9.254:34919 was elected leader, sending a full tablet report...
I20260929 11:33:01.877041 8425 ts_tablet_manager.cc:1428] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 11:33:01.883684 8424 raft_consensus.cc:348] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.884047 8424 raft_consensus.cc:374] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:01.884142 8424 raft_consensus.cc:729] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cceb281db874057995c3ea5a56e7127, State: Initialized, Role: FOLLOWER
I20260929 11:33:01.884743 8424 consensus_queue.cc:260] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.886163 8424 ts_tablet_manager.cc:1428] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 11:33:01.946382 8429 raft_consensus.cc:482] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:01.946672 8429 raft_consensus.cc:504] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.948254 8429 leader_election.cc:288] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:01.955107 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.948813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:52947 (local address 127.6.9.193:35241)
0929 11:33:01.949009 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.949016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.949471 (+ 455us) server_negotiation.cc:407] Connection header received
0929 11:33:01.949679 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.949685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.949786 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.949887 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:01.950693 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.951560 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.953473 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.954257 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.954380 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.954485 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.954749 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.954882 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:33:01.955201 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.948707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:35241 (local address 127.6.9.194:52947)
0929 11:33:01.949347 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.949366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.949515 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.950088 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.950093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.950128 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:01.950547 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.950557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.951678 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.951683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.953314 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.953326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.954463 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.954470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.954605 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.954629 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.954710 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.954838 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":350,"threads_started":1}
I20260929 11:33:01.955662 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.949753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59461 (local address 127.6.9.195:46271)
0929 11:33:01.949928 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.949933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.950468 (+ 535us) server_negotiation.cc:407] Connection header received
0929 11:33:01.950691 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.950696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.950773 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.950875 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:01.951351 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.952314 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.953989 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.954767 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.954883 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.954985 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:01.955307 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 11:33:01.955434 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:33:01.956182 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.949477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:46271 (local address 127.6.9.194:59461)
0929 11:33:01.950355 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.950372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.950511 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.950889 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.950893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:01.950919 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:01.951197 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.951205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.952447 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.952451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.953841 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:01.953853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:01.954985 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:01.954992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:01.955124 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:01.955149 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:33:01.955841 (+ 692us) client_negotiation.cc:240] Negotiation successful
0929 11:33:01.955960 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":548,"threads_started":1}
I20260929 11:33:01.956431 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14afc8cc25774392a525a40cac75ccc1" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127" is_pre_election: true
I20260929 11:33:01.956982 8076 raft_consensus.cc:2454] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:01.957137 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14afc8cc25774392a525a40cac75ccc1" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9" is_pre_election: true
I20260929 11:33:01.957617 8358 raft_consensus.cc:2454] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:01.957659 8147 leader_election.cc:302] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:01.958083 8429 raft_consensus.cc:2781] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:01.958204 8429 raft_consensus.cc:482] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:01.958307 8429 raft_consensus.cc:3037] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:01.960397 8429 raft_consensus.cc:504] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.960975 8429 leader_election.cc:288] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Requested vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:01.961357 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14afc8cc25774392a525a40cac75ccc1" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127"
I20260929 11:33:01.961397 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14afc8cc25774392a525a40cac75ccc1" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9"
I20260929 11:33:01.961534 8076 raft_consensus.cc:3037] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:01.961556 8358 raft_consensus.cc:3037] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:01.963685 8358 raft_consensus.cc:2454] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:01.963915 8076 raft_consensus.cc:2454] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:01.964022 8147 leader_election.cc:302] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6f8dcff5f445b8910fa552dee5e178, 7aa7571192eb43a6b5512590d223a8b9; no voters:
I20260929 11:33:01.964282 8429 raft_consensus.cc:2781] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:01.964720 8429 raft_consensus.cc:686] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Running, Role: LEADER
I20260929 11:33:01.965159 8429 consensus_queue.cc:237] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:01.969614 7918 catalog_manager.cc:5495] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } health_report { overall_health: UNKNOWN } } }
W20260929 11:33:01.991195 8266 tablet.cc:2369] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:33:01.991195 8127 tablet.cc:2369] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:33:01.993181 8405 tablet.cc:2369] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:33:02 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:02.011897 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.998964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.6.9.254:34919)
0929 11:33:01.999115 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:01.999119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:01.999643 (+ 524us) server_negotiation.cc:407] Connection header received
0929 11:33:01.999782 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:01.999786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:01.999854 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:01.999932 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:02.001029 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.001813 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.003090 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.003545 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.003643 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.006833 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:02.006838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:02.006841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:02.006871 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:02.009328 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:02.010011 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:02.010016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:02.010018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:02.010163 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:02.010487 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:02.010491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:02.010493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:02.010869 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:02.010977 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:02.011446 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 11:33:02.011619 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:33:02.011945 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:01.999003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:34919 (local address 127.0.0.1:53134)
0929 11:33:01.999548 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:01.999563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:01.999690 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:01.999962 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:01.999965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:02.000568 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:02.000886 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.000896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.001925 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.001929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.002954 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.002965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.003700 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.003704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.003824 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.004418 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:02.004434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:02.006656 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:02.009455 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:02.009466 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:02.009472 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:02.009882 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:02.010267 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:02.010271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:02.010273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:02.010405 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:02.010998 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:02.011005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:02.011163 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:33:02.011298 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:33:02.011521 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
I20260929 11:33:02.017325 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:53134:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:33:02.018417 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:02.025066 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 9127daf457224b639d7aa145753229cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ebac28e81a84ff0809cc219974e4ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:02.025177 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 9127daf457224b639d7aa145753229cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ebac28e81a84ff0809cc219974e4ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:02.025363 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 9127daf457224b639d7aa145753229cf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ebac28e81a84ff0809cc219974e4ce1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:02.025924 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9127daf457224b639d7aa145753229cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.026162 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9127daf457224b639d7aa145753229cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.026162 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9127daf457224b639d7aa145753229cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.031451 8424 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: Bootstrap starting.
I20260929 11:33:02.033144 8424 tablet_bootstrap.cc:654] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.033218 8425 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: Bootstrap starting.
I20260929 11:33:02.034437 8423 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: Bootstrap starting.
I20260929 11:33:02.034844 8425 tablet_bootstrap.cc:654] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.035327 8424 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: No bootstrap required, opened a new log
I20260929 11:33:02.035432 8424 ts_tablet_manager.cc:1397] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:33:02.036033 8423 tablet_bootstrap.cc:654] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.036095 8424 raft_consensus.cc:348] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.036345 8424 raft_consensus.cc:374] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.036434 8424 raft_consensus.cc:729] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cceb281db874057995c3ea5a56e7127, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.036746 8424 consensus_queue.cc:260] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.037212 8431 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.037480 8425 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: No bootstrap required, opened a new log
I20260929 11:33:02.037591 8431 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.037618 8425 ts_tablet_manager.cc:1397] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:33:02.038048 8424 ts_tablet_manager.cc:1428] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:02.038349 8425 raft_consensus.cc:348] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.038537 8425 raft_consensus.cc:374] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.038620 8425 raft_consensus.cc:729] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa7571192eb43a6b5512590d223a8b9, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.038843 8425 consensus_queue.cc:260] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.039250 8430 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.039609 8430 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.039701 8425 ts_tablet_manager.cc:1428] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:02.046036 8423 tablet_bootstrap.cc:492] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: No bootstrap required, opened a new log
I20260929 11:33:02.046175 8423 ts_tablet_manager.cc:1397] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20260929 11:33:02.046799 8423 raft_consensus.cc:348] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.046986 8423 raft_consensus.cc:374] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.047049 8423 raft_consensus.cc:729] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.047264 8423 consensus_queue.cc:260] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.047870 8423 ts_tablet_manager.cc:1428] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 11:33:02.047658 8429 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.048053 8429 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.053500 8429 raft_consensus.cc:482] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:02.053650 8429 raft_consensus.cc:504] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.054293 8429 leader_election.cc:288] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:02.054711 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9127daf457224b639d7aa145753229cf" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127" is_pre_election: true
I20260929 11:33:02.054739 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9127daf457224b639d7aa145753229cf" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9" is_pre_election: true
I20260929 11:33:02.054992 8358 raft_consensus.cc:2454] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:02.054989 8076 raft_consensus.cc:2454] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:02.055339 8147 leader_election.cc:302] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:02.055575 8429 raft_consensus.cc:2781] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:02.055660 8429 raft_consensus.cc:482] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:02.055756 8429 raft_consensus.cc:3037] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.057719 8429 raft_consensus.cc:504] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.058194 8429 leader_election.cc:288] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Requested vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:02.058630 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9127daf457224b639d7aa145753229cf" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127"
I20260929 11:33:02.058686 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9127daf457224b639d7aa145753229cf" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9"
I20260929 11:33:02.058820 8076 raft_consensus.cc:3037] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.058928 8358 raft_consensus.cc:3037] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.060830 8076 raft_consensus.cc:2454] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:02.060887 8358 raft_consensus.cc:2454] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:02.061282 8147 leader_election.cc:302] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:02.061564 8429 raft_consensus.cc:2781] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:02.061754 8429 raft_consensus.cc:686] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Running, Role: LEADER
I20260929 11:33:02.062088 8429 consensus_queue.cc:237] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.063735 8434 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: TxnStatusTablet state changed. Reason: New leader 4c6f8dcff5f445b8910fa552dee5e178. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.064106 8434 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:02.064839 8446 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:33:02.065109 8446 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:33:02.065196 7918 catalog_manager.cc:5495] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:02.085196 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:02.079038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40891 (local address 127.0.0.1:47838)
0929 11:33:02.079265 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:02.079285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:02.079411 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:02.079931 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:02.079934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:02.079944 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:02.080110 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.080117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.081553 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.081556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.082524 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.082535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.083460 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.083464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.083612 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.083665 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:02.084696 (+ 1031us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:02.084714 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 11:33:02.084803 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:33:02.084960 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:33:02.085235 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:02.079219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.6.9.194:40891)
0929 11:33:02.079406 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:02.079413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:02.079434 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:33:02.079607 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:02.079613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:02.079766 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:02.079917 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:02.080234 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.081440 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.082667 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.083317 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.083451 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.083861 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:02.084573 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:02.084749 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:02.084834 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 11:33:02.084922 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:33:02.093865 8358 raft_consensus.cc:1261] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.093951 8076 raft_consensus.cc:1261] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.094870 8434 consensus_queue.cc:1035] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:02.095216 8429 consensus_queue.cc:1035] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:02.110059 8430 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: TxnStatusTablet state changed. Reason: New leader 4c6f8dcff5f445b8910fa552dee5e178. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.110401 8430 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.113859 8449 mvcc.cc:205] Tried to move back new op lower bound from 7334631760246575104 to 7334631760128806912. Current Snapshot: MvccSnapshot[applied={T|T < 7334631760246575104}]
I20260929 11:33:02.114741 8434 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.115029 8434 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:02.116230 8431 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: TxnStatusTablet state changed. Reason: New leader 4c6f8dcff5f445b8910fa552dee5e178. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.116501 8431 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.116811 8450 mvcc.cc:205] Tried to move back new op lower bound from 7334631760246575104 to 7334631760128806912. Current Snapshot: MvccSnapshot[applied={T|T < 7334631760246575104}]
I20260929 11:33:02.118644 8429 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.118983 8429 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 4c6f8dcff5f445b8910fa552dee5e178: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:02.132254 8430 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.132536 8430 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 7aa7571192eb43a6b5512590d223a8b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.142372 8431 tablet_replica.cc:439] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } } }
I20260929 11:33:02.142663 8431 tablet_replica.cc:442] T 9127daf457224b639d7aa145753229cf P 0cceb281db874057995c3ea5a56e7127: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:02.149897 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:02.142331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.6.9.194:40891)
0929 11:33:02.142531 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:02.142537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:02.143871 (+ 1334us) server_negotiation.cc:407] Connection header received
0929 11:33:02.143952 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:02.143956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:02.144045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:02.144130 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:02.144741 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.145892 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.147929 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.148376 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.148480 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.149089 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:02.149236 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:02.149333 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:02.149517 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 11:33:02.149600 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:33:02.174149 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:02.160772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:45363 (local address 127.6.9.194:40891)
0929 11:33:02.160984 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:02.160990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:02.169355 (+ 8365us) server_negotiation.cc:407] Connection header received
0929 11:33:02.169549 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:02.169553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:02.169611 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:02.169679 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:02.170170 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.170909 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.172304 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.172790 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.172904 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.173376 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:02.173473 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:02.173546 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:02.173694 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 11:33:02.173818 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260929 11:33:02.174250 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:02.160652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40891 (local address 127.6.9.194:45363)
0929 11:33:02.169234 (+ 8582us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:02.169253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:02.169424 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:02.169704 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:02.169708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:02.169719 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:02.170030 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.170040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.171037 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.171041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.172176 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:02.172188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:02.172964 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:02.172970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:02.173122 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:02.173181 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:02.173588 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:02.173596 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:33:02.173666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 11:33:02.173884 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8358,"thread_start_us":8202,"threads_started":1}
I20260929 11:33:02.192992 8076 raft_consensus.cc:1261] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.193373 8358 raft_consensus.cc:1261] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.193753 8434 consensus_queue.cc:1035] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:02.194449 8461 consensus_queue.cc:1035] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:02.268975 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:02.269284 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:02.273185 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Sending DeleteTablet for 3 replicas of tablet 14afc8cc25774392a525a40cac75ccc1
I20260929 11:33:02.274355 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet 14afc8cc25774392a525a40cac75ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:02 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47826
I20260929 11:33:02.275650 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 14afc8cc25774392a525a40cac75ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:02 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:34684
I20260929 11:33:02.275151 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 14afc8cc25774392a525a40cac75ccc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:02 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60976
I20260929 11:33:02.276332 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:02.276978 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:02.281662 8468 tablet_replica.cc:330] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: stopping tablet replica
I20260929 11:33:02.282209 8468 raft_consensus.cc:2229] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:02.282980 8468 raft_consensus.cc:2258] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:02.283699 8469 tablet_replica.cc:330] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: stopping tablet replica
I20260929 11:33:02.284224 8470 tablet_replica.cc:330] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: stopping tablet replica
I20260929 11:33:02.284708 8469 raft_consensus.cc:2229] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:02.285181 8470 raft_consensus.cc:2229] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:02.285606 8469 raft_consensus.cc:2258] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:02.285987 8470 raft_consensus.cc:2258] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:02.287017 8195 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 (DEFAULT_TABLE table=test-table [id=14204f8987fa41d7a52052af846383c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:02.287494 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbbf7a224f34c61a372f99f84f570a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.287882 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 (DEFAULT_TABLE table=test-table [id=14204f8987fa41d7a52052af846383c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:02.288331 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbbf7a224f34c61a372f99f84f570a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.288465 8469 ts_tablet_manager.cc:1905] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:02.293068 8334 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 (DEFAULT_TABLE table=test-table [id=14204f8987fa41d7a52052af846383c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:02.293546 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fbbf7a224f34c61a372f99f84f570a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:02.296022 8469 ts_tablet_manager.cc:1918] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:02.296309 8469 log.cc:1196] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/14afc8cc25774392a525a40cac75ccc1
I20260929 11:33:02.296988 8469 ts_tablet_manager.cc:1939] T 14afc8cc25774392a525a40cac75ccc1 P 7aa7571192eb43a6b5512590d223a8b9: Deleting consensus metadata
I20260929 11:33:02.297334 8423 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Bootstrap starting.
I20260929 11:33:02.299225 8423 tablet_bootstrap.cc:654] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.299283 7904 catalog_manager.cc:4841] TS 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271): tablet 14afc8cc25774392a525a40cac75ccc1 (table test-table [id=273c28247a4747a3a1d1074945ad7e03]) successfully deleted
I20260929 11:33:02.301525 8468 ts_tablet_manager.cc:1905] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:02.301736 8423 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: No bootstrap required, opened a new log
I20260929 11:33:02.301950 8423 ts_tablet_manager.cc:1397] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 11:33:02.302590 8423 raft_consensus.cc:348] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.302788 8423 raft_consensus.cc:374] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.302918 8423 raft_consensus.cc:729] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.303180 8423 consensus_queue.cc:260] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.305050 8423 ts_tablet_manager.cc:1428] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 11:33:02.311273 8425 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Bootstrap starting.
I20260929 11:33:02.311524 8424 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Bootstrap starting.
I20260929 11:33:02.312764 8425 tablet_bootstrap.cc:654] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.313213 8424 tablet_bootstrap.cc:654] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:02.314980 8425 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: No bootstrap required, opened a new log
I20260929 11:33:02.315138 8425 ts_tablet_manager.cc:1397] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 11:33:02.315812 8425 raft_consensus.cc:348] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.316043 8425 raft_consensus.cc:374] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.316118 8425 raft_consensus.cc:729] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa7571192eb43a6b5512590d223a8b9, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.316386 8425 consensus_queue.cc:260] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.317471 8425 ts_tablet_manager.cc:1428] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:33:02.319363 8470 ts_tablet_manager.cc:1905] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:02.321908 8424 tablet_bootstrap.cc:492] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: No bootstrap required, opened a new log
I20260929 11:33:02.322098 8424 ts_tablet_manager.cc:1397] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 11:33:02.322731 8424 raft_consensus.cc:348] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.322937 8424 raft_consensus.cc:374] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:02.323016 8424 raft_consensus.cc:729] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cceb281db874057995c3ea5a56e7127, State: Initialized, Role: FOLLOWER
I20260929 11:33:02.323247 8424 consensus_queue.cc:260] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.323915 8424 ts_tablet_manager.cc:1428] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 11:33:02.327373 8468 ts_tablet_manager.cc:1918] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:02.327456 8470 ts_tablet_manager.cc:1918] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:02.327579 8468 log.cc:1196] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/14afc8cc25774392a525a40cac75ccc1
I20260929 11:33:02.327610 8470 log.cc:1196] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/14afc8cc25774392a525a40cac75ccc1
I20260929 11:33:02.328135 8470 ts_tablet_manager.cc:1939] T 14afc8cc25774392a525a40cac75ccc1 P 0cceb281db874057995c3ea5a56e7127: Deleting consensus metadata
I20260929 11:33:02.328166 8468 ts_tablet_manager.cc:1939] T 14afc8cc25774392a525a40cac75ccc1 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting consensus metadata
I20260929 11:33:02.329838 7903 catalog_manager.cc:4841] TS 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194:40891): tablet 14afc8cc25774392a525a40cac75ccc1 (table test-table [id=273c28247a4747a3a1d1074945ad7e03]) successfully deleted
I20260929 11:33:02.330129 7904 catalog_manager.cc:4841] TS 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241): tablet 14afc8cc25774392a525a40cac75ccc1 (table test-table [id=273c28247a4747a3a1d1074945ad7e03]) successfully deleted
I20260929 11:33:02.331487 8434 raft_consensus.cc:482] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:02.331665 8434 raft_consensus.cc:504] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.332429 8434 leader_election.cc:288] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:02.332877 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbbf7a224f34c61a372f99f84f570a2" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127" is_pre_election: true
I20260929 11:33:02.333117 8076 raft_consensus.cc:2454] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:02.333065 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbbf7a224f34c61a372f99f84f570a2" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9" is_pre_election: true
I20260929 11:33:02.333302 8357 raft_consensus.cc:2454] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:02.333554 8147 leader_election.cc:302] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:02.333841 8434 raft_consensus.cc:2781] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:02.333966 8434 raft_consensus.cc:482] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:02.334057 8434 raft_consensus.cc:3037] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.337169 8434 raft_consensus.cc:504] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.337805 8434 leader_election.cc:288] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Requested vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:02.338290 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbbf7a224f34c61a372f99f84f570a2" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127"
I20260929 11:33:02.338348 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbbf7a224f34c61a372f99f84f570a2" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9"
I20260929 11:33:02.338519 8076 raft_consensus.cc:3037] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.338541 8357 raft_consensus.cc:3037] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:02.340543 8076 raft_consensus.cc:2454] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:02.340967 8147 leader_election.cc:302] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:02.341393 8357 raft_consensus.cc:2454] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:02.341727 8434 raft_consensus.cc:2781] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:02.341894 8434 raft_consensus.cc:686] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Running, Role: LEADER
I20260929 11:33:02.342170 8434 consensus_queue.cc:237] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:02.344993 7919 catalog_manager.cc:5495] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:02.382148 8357 raft_consensus.cc:1261] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.382532 8076 raft_consensus.cc:1261] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:02.382838 8465 consensus_queue.cc:1035] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:02.383143 8464 consensus_queue.cc:1035] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:33:02.408618 8449 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:33:03.994314 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:03.994552 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:03.998956 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Sending DeleteTablet for 3 replicas of tablet 4fbbf7a224f34c61a372f99f84f570a2
I20260929 11:33:03.999852 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:03 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47826
I20260929 11:33:03.999874 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:03 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:34684
I20260929 11:33:03.999908 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fbbf7a224f34c61a372f99f84f570a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:03 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60976
I20260929 11:33:04.005259 8498 tablet_replica.cc:330] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: stopping tablet replica
I20260929 11:33:04.005515 8498 raft_consensus.cc:2229] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:04.005800 8498 raft_consensus.cc:2258] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.006388 8497 tablet_replica.cc:330] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: stopping tablet replica
I20260929 11:33:04.006567 8497 raft_consensus.cc:2229] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:04.006915 8497 raft_consensus.cc:2258] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.007464 8499 tablet_replica.cc:330] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: stopping tablet replica
I20260929 11:33:04.007705 8499 raft_consensus.cc:2229] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:04.007974 8499 raft_consensus.cc:2258] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.007997 8498 ts_tablet_manager.cc:1905] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:04.008036 8497 ts_tablet_manager.cc:1905] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:04.008940 8499 ts_tablet_manager.cc:1905] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:04.014971 8497 ts_tablet_manager.cc:1918] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:04.015070 8498 ts_tablet_manager.cc:1918] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:04.015266 8498 log.cc:1196] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/4fbbf7a224f34c61a372f99f84f570a2
I20260929 11:33:04.015266 8497 log.cc:1196] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/4fbbf7a224f34c61a372f99f84f570a2
I20260929 11:33:04.015461 8499 ts_tablet_manager.cc:1918] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:04.015636 8499 log.cc:1196] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/4fbbf7a224f34c61a372f99f84f570a2
I20260929 11:33:04.015792 8498 ts_tablet_manager.cc:1939] T 4fbbf7a224f34c61a372f99f84f570a2 P 7aa7571192eb43a6b5512590d223a8b9: Deleting consensus metadata
I20260929 11:33:04.015892 8497 ts_tablet_manager.cc:1939] T 4fbbf7a224f34c61a372f99f84f570a2 P 4c6f8dcff5f445b8910fa552dee5e178: Deleting consensus metadata
I20260929 11:33:04.016072 8499 ts_tablet_manager.cc:1939] T 4fbbf7a224f34c61a372f99f84f570a2 P 0cceb281db874057995c3ea5a56e7127: Deleting consensus metadata
I20260929 11:33:04.017099 7904 catalog_manager.cc:4841] TS 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271): tablet 4fbbf7a224f34c61a372f99f84f570a2 (table test-table [id=14204f8987fa41d7a52052af846383c7]) successfully deleted
I20260929 11:33:04.017151 7903 catalog_manager.cc:4841] TS 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194:40891): tablet 4fbbf7a224f34c61a372f99f84f570a2 (table test-table [id=14204f8987fa41d7a52052af846383c7]) successfully deleted
I20260929 11:33:04.017407 7904 catalog_manager.cc:4841] TS 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241): tablet 4fbbf7a224f34c61a372f99f84f570a2 (table test-table [id=14204f8987fa41d7a52052af846383c7]) successfully deleted
Sep 29 11:33:04 dist-test-slave-c0l9 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:33:04 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:04.057081 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:04.037675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.6.9.254:34919)
0929 11:33:04.038218 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:04.038223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:04.038245 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:33:04.038386 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:04.038390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:04.038452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:04.038542 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:04.039783 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:04.040533 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:04.041726 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:04.042135 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:04.042235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:04.051686 (+ 9451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:04.051693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:04.051697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:04.051739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:04.054600 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:04.055220 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:04.055223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:04.055226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:04.055366 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:04.055788 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:04.055793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:04.055797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:04.056123 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:04.056312 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:04.056604 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 11:33:04.056822 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":273,"threads_started":1}
I20260929 11:33:04.060463 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:04.061107 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:04.067425 8055 tablet_service.cc:1467] Processing CreateTablet for tablet d9141edaccc44c2d88b2e151eae8348b (DEFAULT_TABLE table=test-table [id=17b17e220b32416ea02cfa050ff6ccd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.067498 8195 tablet_service.cc:1467] Processing CreateTablet for tablet d9141edaccc44c2d88b2e151eae8348b (DEFAULT_TABLE table=test-table [id=17b17e220b32416ea02cfa050ff6ccd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.067740 8334 tablet_service.cc:1467] Processing CreateTablet for tablet d9141edaccc44c2d88b2e151eae8348b (DEFAULT_TABLE table=test-table [id=17b17e220b32416ea02cfa050ff6ccd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.067906 8195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9141edaccc44c2d88b2e151eae8348b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.067926 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9141edaccc44c2d88b2e151eae8348b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.068116 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9141edaccc44c2d88b2e151eae8348b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.074477 8508 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Bootstrap starting.
I20260929 11:33:04.075058 8510 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Bootstrap starting.
I20260929 11:33:04.075748 8509 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Bootstrap starting.
I20260929 11:33:04.076201 8508 tablet_bootstrap.cc:654] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.076936 8510 tablet_bootstrap.cc:654] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.077268 8509 tablet_bootstrap.cc:654] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.078647 8508 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: No bootstrap required, opened a new log
I20260929 11:33:04.078809 8508 ts_tablet_manager.cc:1397] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 11:33:04.079172 8509 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: No bootstrap required, opened a new log
I20260929 11:33:04.079295 8509 ts_tablet_manager.cc:1397] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 11:33:04.079604 8508 raft_consensus.cc:348] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.079864 8508 raft_consensus.cc:374] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.079953 8508 raft_consensus.cc:729] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa7571192eb43a6b5512590d223a8b9, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.079892 8509 raft_consensus.cc:348] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.080077 8509 raft_consensus.cc:374] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.080153 8509 raft_consensus.cc:729] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.080185 8508 consensus_queue.cc:260] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.080353 8509 consensus_queue.cc:260] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.082064 8508 ts_tablet_manager.cc:1428] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260929 11:33:04.082804 8509 ts_tablet_manager.cc:1428] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:33:04.083801 8510 tablet_bootstrap.cc:492] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: No bootstrap required, opened a new log
I20260929 11:33:04.083933 8510 ts_tablet_manager.cc:1397] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:33:04.084638 8510 raft_consensus.cc:348] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.084903 8510 raft_consensus.cc:374] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.085001 8510 raft_consensus.cc:729] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cceb281db874057995c3ea5a56e7127, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.085224 8510 consensus_queue.cc:260] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.085970 8510 ts_tablet_manager.cc:1428] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:33:04.359839 8464 raft_consensus.cc:482] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:04.360117 8464 raft_consensus.cc:504] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.360791 8464 leader_election.cc:288] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:04.361632 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9141edaccc44c2d88b2e151eae8348b" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127" is_pre_election: true
I20260929 11:33:04.361696 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9141edaccc44c2d88b2e151eae8348b" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9" is_pre_election: true
I20260929 11:33:04.361915 8076 raft_consensus.cc:2454] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:04.361928 8358 raft_consensus.cc:2454] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:04.362461 8147 leader_election.cc:302] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6f8dcff5f445b8910fa552dee5e178, 7aa7571192eb43a6b5512590d223a8b9; no voters:
I20260929 11:33:04.362778 8464 raft_consensus.cc:2781] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:04.362906 8464 raft_consensus.cc:482] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:04.363008 8464 raft_consensus.cc:3037] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.365134 8464 raft_consensus.cc:504] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.365676 8464 leader_election.cc:288] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Requested vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:04.366179 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9141edaccc44c2d88b2e151eae8348b" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127"
I20260929 11:33:04.366274 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9141edaccc44c2d88b2e151eae8348b" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9"
I20260929 11:33:04.366372 8076 raft_consensus.cc:3037] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.366477 8358 raft_consensus.cc:3037] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.368279 8076 raft_consensus.cc:2454] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:04.368351 8358 raft_consensus.cc:2454] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:04.368640 8147 leader_election.cc:302] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:04.368930 8464 raft_consensus.cc:2781] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:04.369073 8464 raft_consensus.cc:686] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Running, Role: LEADER
I20260929 11:33:04.369326 8464 consensus_queue.cc:237] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.371862 7918 catalog_manager.cc:5495] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:04.452133 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:04.447271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.6.9.194:40891)
0929 11:33:04.447781 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:04.447785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:04.447802 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:33:04.447851 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:04.447855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:04.447914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:04.447995 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:04.448399 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:04.449169 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:04.450458 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:04.450865 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:04.450972 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:04.451408 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:04.451527 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:04.451601 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:04.451750 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 11:33:04.451827 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260929 11:33:04.461023 8358 raft_consensus.cc:1261] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:04.461117 8076 raft_consensus.cc:1261] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:04.461596 8464 consensus_queue.cc:1035] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:04.461850 8484 consensus_queue.cc:1035] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:04.504891 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:04.505132 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:04.507979 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Sending DeleteTablet for 3 replicas of tablet d9141edaccc44c2d88b2e151eae8348b
I20260929 11:33:04.508919 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet d9141edaccc44c2d88b2e151eae8348b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:04 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:34684
I20260929 11:33:04.508916 8194 tablet_service.cc:1514] Processing DeleteTablet for tablet d9141edaccc44c2d88b2e151eae8348b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:04 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47826
I20260929 11:33:04.508922 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet d9141edaccc44c2d88b2e151eae8348b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:04 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60976
I20260929 11:33:04.509254 8497 tablet_replica.cc:330] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: stopping tablet replica
I20260929 11:33:04.509279 8499 tablet_replica.cc:330] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: stopping tablet replica
I20260929 11:33:04.509471 8499 raft_consensus.cc:2229] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:04.509471 8497 raft_consensus.cc:2229] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:04.509672 8499 raft_consensus.cc:2258] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.509805 8497 raft_consensus.cc:2258] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.509975 8498 tablet_replica.cc:330] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: stopping tablet replica
I20260929 11:33:04.510171 8498 raft_consensus.cc:2229] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:04.510378 8498 raft_consensus.cc:2258] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:04.510804 8499 ts_tablet_manager.cc:1905] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:04.510974 8497 ts_tablet_manager.cc:1905] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:04.509994 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 11:33:04.511377 8498 ts_tablet_manager.cc:1905] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 11:33:04.511874 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:04.517254 8499 ts_tablet_manager.cc:1918] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:04.517406 8497 ts_tablet_manager.cc:1918] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:04.517464 8499 log.cc:1196] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/d9141edaccc44c2d88b2e151eae8348b
I20260929 11:33:04.517649 8497 log.cc:1196] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/d9141edaccc44c2d88b2e151eae8348b
I20260929 11:33:04.517947 8499 ts_tablet_manager.cc:1939] T d9141edaccc44c2d88b2e151eae8348b P 0cceb281db874057995c3ea5a56e7127: Deleting consensus metadata
I20260929 11:33:04.517967 8498 ts_tablet_manager.cc:1918] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:04.518143 8498 log.cc:1196] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/d9141edaccc44c2d88b2e151eae8348b
I20260929 11:33:04.518492 8497 ts_tablet_manager.cc:1939] T d9141edaccc44c2d88b2e151eae8348b P 4c6f8dcff5f445b8910fa552dee5e178: Deleting consensus metadata
I20260929 11:33:04.518527 8498 ts_tablet_manager.cc:1939] T d9141edaccc44c2d88b2e151eae8348b P 7aa7571192eb43a6b5512590d223a8b9: Deleting consensus metadata
I20260929 11:33:04.519475 7904 catalog_manager.cc:4841] TS 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241): tablet d9141edaccc44c2d88b2e151eae8348b (table test-table [id=17b17e220b32416ea02cfa050ff6ccd8]) successfully deleted
I20260929 11:33:04.519677 7903 catalog_manager.cc:4841] TS 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194:40891): tablet d9141edaccc44c2d88b2e151eae8348b (table test-table [id=17b17e220b32416ea02cfa050ff6ccd8]) successfully deleted
I20260929 11:33:04.519727 7904 catalog_manager.cc:4841] TS 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271): tablet d9141edaccc44c2d88b2e151eae8348b (table test-table [id=17b17e220b32416ea02cfa050ff6ccd8]) successfully deleted
I20260929 11:33:04.519850 8334 tablet_service.cc:1467] Processing CreateTablet for tablet dbad7c3e967c491b98007f6936e9c64e (DEFAULT_TABLE table=test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.519928 8194 tablet_service.cc:1467] Processing CreateTablet for tablet dbad7c3e967c491b98007f6936e9c64e (DEFAULT_TABLE table=test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.520270 8334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbad7c3e967c491b98007f6936e9c64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.520360 8194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbad7c3e967c491b98007f6936e9c64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.520387 8055 tablet_service.cc:1467] Processing CreateTablet for tablet dbad7c3e967c491b98007f6936e9c64e (DEFAULT_TABLE table=test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:04.520776 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbad7c3e967c491b98007f6936e9c64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:04.525480 8509 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Bootstrap starting.
I20260929 11:33:04.526194 8508 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Bootstrap starting.
I20260929 11:33:04.526726 8510 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Bootstrap starting.
I20260929 11:33:04.526971 8509 tablet_bootstrap.cc:654] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.527717 8508 tablet_bootstrap.cc:654] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.527966 8510 tablet_bootstrap.cc:654] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:04.529192 8509 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: No bootstrap required, opened a new log
I20260929 11:33:04.529382 8509 ts_tablet_manager.cc:1397] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 11:33:04.530000 8509 raft_consensus.cc:348] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.530274 8509 raft_consensus.cc:374] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.530375 8509 raft_consensus.cc:729] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.530561 8508 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: No bootstrap required, opened a new log
I20260929 11:33:04.530723 8508 ts_tablet_manager.cc:1397] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 11:33:04.530624 8509 consensus_queue.cc:260] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.531117 8509 ts_tablet_manager.cc:1428] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:04.531454 8508 raft_consensus.cc:348] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.531692 8508 raft_consensus.cc:374] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.531762 8508 raft_consensus.cc:729] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa7571192eb43a6b5512590d223a8b9, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.532217 8508 consensus_queue.cc:260] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.533388 8508 ts_tablet_manager.cc:1428] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 11:33:04.535468 8510 tablet_bootstrap.cc:492] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: No bootstrap required, opened a new log
I20260929 11:33:04.535655 8510 ts_tablet_manager.cc:1397] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 11:33:04.536403 8510 raft_consensus.cc:348] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.536625 8510 raft_consensus.cc:374] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:04.536736 8510 raft_consensus.cc:729] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cceb281db874057995c3ea5a56e7127, State: Initialized, Role: FOLLOWER
I20260929 11:33:04.536957 8510 consensus_queue.cc:260] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.537420 8510 ts_tablet_manager.cc:1428] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:04.554886 8464 raft_consensus.cc:482] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:04.555109 8464 raft_consensus.cc:504] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.555694 8464 leader_election.cc:288] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:04.556165 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbad7c3e967c491b98007f6936e9c64e" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127" is_pre_election: true
I20260929 11:33:04.556232 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbad7c3e967c491b98007f6936e9c64e" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9" is_pre_election: true
I20260929 11:33:04.556393 8076 raft_consensus.cc:2454] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:04.556437 8357 raft_consensus.cc:2454] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 0.
I20260929 11:33:04.556816 8147 leader_election.cc:302] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:04.557073 8464 raft_consensus.cc:2781] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:04.557174 8464 raft_consensus.cc:482] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:04.557309 8464 raft_consensus.cc:3037] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.559283 8464 raft_consensus.cc:504] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.559779 8464 leader_election.cc:288] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Requested vote from peers 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241), 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271)
I20260929 11:33:04.560186 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbad7c3e967c491b98007f6936e9c64e" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cceb281db874057995c3ea5a56e7127"
I20260929 11:33:04.560240 8357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbad7c3e967c491b98007f6936e9c64e" candidate_uuid: "4c6f8dcff5f445b8910fa552dee5e178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa7571192eb43a6b5512590d223a8b9"
I20260929 11:33:04.560339 8076 raft_consensus.cc:3037] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.560422 8357 raft_consensus.cc:3037] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:04.562801 8076 raft_consensus.cc:2454] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:04.562889 8357 raft_consensus.cc:2454] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6f8dcff5f445b8910fa552dee5e178 in term 1.
I20260929 11:33:04.563247 8147 leader_election.cc:302] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cceb281db874057995c3ea5a56e7127, 4c6f8dcff5f445b8910fa552dee5e178; no voters:
I20260929 11:33:04.563520 8464 raft_consensus.cc:2781] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:04.563689 8464 raft_consensus.cc:686] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6f8dcff5f445b8910fa552dee5e178, State: Running, Role: LEADER
I20260929 11:33:04.563941 8464 consensus_queue.cc:237] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } }
I20260929 11:33:04.566622 7919 catalog_manager.cc:5495] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "4c6f8dcff5f445b8910fa552dee5e178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6f8dcff5f445b8910fa552dee5e178" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:04.594501 8357 raft_consensus.cc:1261] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:04.594542 8076 raft_consensus.cc:1261] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Refusing update from remote peer 4c6f8dcff5f445b8910fa552dee5e178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:04.594986 8464 consensus_queue.cc:1035] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cceb281db874057995c3ea5a56e7127" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:04.595234 8484 consensus_queue.cc:1035] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aa7571192eb43a6b5512590d223a8b9" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:33:04.613212 8477 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:33:06.193284 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:06.193580 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:06.198228 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ffe1ae3fa954694a958bea6deb52ec5: Sending DeleteTablet for 3 replicas of tablet dbad7c3e967c491b98007f6936e9c64e
I20260929 11:33:06.199093 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet dbad7c3e967c491b98007f6936e9c64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:34684
I20260929 11:33:06.199093 8195 tablet_service.cc:1514] Processing DeleteTablet for tablet dbad7c3e967c491b98007f6936e9c64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47826
I20260929 11:33:06.199522 8334 tablet_service.cc:1514] Processing DeleteTablet for tablet dbad7c3e967c491b98007f6936e9c64e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60976
I20260929 11:33:06.199755 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7990
I20260929 11:33:06.200094 8549 tablet_replica.cc:330] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: stopping tablet replica
I20260929 11:33:06.200294 8551 tablet_replica.cc:330] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: stopping tablet replica
I20260929 11:33:06.200297 8549 raft_consensus.cc:2229] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:06.200479 8551 raft_consensus.cc:2229] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:06.200690 8549 raft_consensus.cc:2258] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:06.200734 8551 raft_consensus.cc:2258] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:06.201004 8550 tablet_replica.cc:330] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: stopping tablet replica
I20260929 11:33:06.201175 8550 raft_consensus.cc:2229] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:06.201434 8550 raft_consensus.cc:2258] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:06.202185 8551 ts_tablet_manager.cc:1905] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:06.202493 8550 ts_tablet_manager.cc:1905] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:06.205865 8549 ts_tablet_manager.cc:1905] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:06.208882 8551 ts_tablet_manager.cc:1918] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:06.209028 8551 log.cc:1196] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/dbad7c3e967c491b98007f6936e9c64e
I20260929 11:33:06.209499 8551 ts_tablet_manager.cc:1939] T dbad7c3e967c491b98007f6936e9c64e P 7aa7571192eb43a6b5512590d223a8b9: Deleting consensus metadata
I20260929 11:33:06.209694 8550 ts_tablet_manager.cc:1918] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:06.209848 8550 log.cc:1196] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/dbad7c3e967c491b98007f6936e9c64e
I20260929 11:33:06.210311 8550 ts_tablet_manager.cc:1939] T dbad7c3e967c491b98007f6936e9c64e P 0cceb281db874057995c3ea5a56e7127: Deleting consensus metadata
I20260929 11:33:06.210722 7904 catalog_manager.cc:4841] TS 7aa7571192eb43a6b5512590d223a8b9 (127.6.9.195:46271): tablet dbad7c3e967c491b98007f6936e9c64e (table test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]) successfully deleted
I20260929 11:33:06.211540 7904 catalog_manager.cc:4841] TS 0cceb281db874057995c3ea5a56e7127 (127.6.9.193:35241): tablet dbad7c3e967c491b98007f6936e9c64e (table test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]) successfully deleted
I20260929 11:33:06.214892 8549 ts_tablet_manager.cc:1918] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:33:06.215031 8549 log.cc:1196] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/dbad7c3e967c491b98007f6936e9c64e
I20260929 11:33:06.215376 8549 ts_tablet_manager.cc:1939] T dbad7c3e967c491b98007f6936e9c64e P 4c6f8dcff5f445b8910fa552dee5e178: Deleting consensus metadata
I20260929 11:33:06.216362 7903 catalog_manager.cc:4841] TS 4c6f8dcff5f445b8910fa552dee5e178 (127.6.9.194:40891): tablet dbad7c3e967c491b98007f6936e9c64e (table test-table [id=5f42b4b46bae4d19ad3aa9669cfc77e3]) successfully deleted
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:06.226670 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:06.200758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.6.9.193:35241)
0929 11:33:06.211357 (+ 10599us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:06.211364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:06.213198 (+ 1834us) server_negotiation.cc:407] Connection header received
0929 11:33:06.213367 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:06.213370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:06.213450 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:06.213539 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:06.215109 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.215900 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.217472 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.217916 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.218030 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:06.221806 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:06.221821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:06.221837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:06.221868 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:06.224324 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.224979 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.224986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.224988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.225147 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.225455 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.225458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.225460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.225763 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:06.225990 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:06.226184 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 11:33:06.226346 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10448,"thread_start_us":10317,"threads_started":1}
W20260929 11:33:06.227324 8121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34688
W20260929 11:33:06.227902 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:06.228087 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7990
I20260929 11:33:06.242103 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8130
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:06.257421 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:06.242828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.6.9.194:40891)
0929 11:33:06.243449 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:06.243454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:06.243468 (+ 14us) server_negotiation.cc:407] Connection header received
0929 11:33:06.243515 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:06.243519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:06.243575 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:06.243674 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:06.244936 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.245708 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.246911 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.247337 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.247444 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:06.251683 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:06.251698 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:06.251715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:06.251750 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:06.254640 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.255300 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.255308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.255310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.255445 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.255791 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.255795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.255797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.256153 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:06.256424 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:06.256907 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 11:33:06.257134 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":359,"threads_started":1}
W20260929 11:33:06.258074 8260 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47856
W20260929 11:33:06.258694 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:06.258834 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8130
I20260929 11:33:06.278980 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8269
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:06.295146 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:06.279743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.6.9.195:46271)
0929 11:33:06.280209 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:06.280214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:06.280229 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:33:06.280282 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:06.280286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:06.280345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:06.280444 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:06.281853 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.282661 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.283864 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:06.284505 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:06.284649 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:06.289521 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:06.289545 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:06.289568 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:06.289621 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:06.292684 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.293396 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.293403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.293405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.293562 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:06.293869 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:06.293871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:06.293873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:06.294200 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:06.294407 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:06.294650 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 11:33:06.294816 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
W20260929 11:33:06.295830 8399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60980
W20260929 11:33:06.296485 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:06.296648 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8269
I20260929 11:33:06.311972 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 7887
I20260929 11:33:06.313165 7959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:06.426142 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 7887
2026-09-29T11:33:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7872 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[8559](info): set up 2 sockets
Sep 29 11:33:06 dist-test-slave-c0l9 krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260929 11:33:07.420686 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:07 dist-test-slave-c0l9 krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:07.593263 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44287
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:07.901180 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:07.901517 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:07.901650 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:07.908882 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:07.909009 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:07.909087 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:07.909161 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:07.909222 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:07.919134 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:07.920460 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:07.921857 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:07.933130 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:07.933154 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:07.933090 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:07.933750 8572 server_base.cc:1034] running on GCE node
I20260929 11:33:07.934487 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:07.935801 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:07.937002 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790681587936956 us; error 31 us; skew 500 ppm
I20260929 11:33:07.937336 8572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:07 dist-test-slave-c0l9 krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681587, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:07.940156 8572 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:07.941501 8572 webserver.cc:460] Webserver started at http://127.6.9.254:37193/ using document root <none> and password file <none>
I20260929 11:33:07.942154 8572 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:07.942271 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:07.942554 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:07.945380 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e"
format_stamp: "Formatted at 2026-09-29 11:33:07 on dist-test-slave-c0l9"
server_key: "618564216926adc22684827d5490bbc1"
server_key_iv: "5edf748b94b3a7e85098bdb93bc7535c"
server_key_version: "encryptionkey@0"
I20260929 11:33:07.946108 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e"
format_stamp: "Formatted at 2026-09-29 11:33:07 on dist-test-slave-c0l9"
server_key: "618564216926adc22684827d5490bbc1"
server_key_iv: "5edf748b94b3a7e85098bdb93bc7535c"
server_key_version: "encryptionkey@0"
I20260929 11:33:07.950819 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 11:33:07.954022 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:07.955214 8572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:33:07.955366 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e"
format_stamp: "Formatted at 2026-09-29 11:33:07 on dist-test-slave-c0l9"
server_key: "618564216926adc22684827d5490bbc1"
server_key_iv: "5edf748b94b3a7e85098bdb93bc7535c"
server_key_version: "encryptionkey@0"
I20260929 11:33:07.955534 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:07.969955 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:07.973958 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:07.974225 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:07.991075 8572 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:44287
I20260929 11:33:07.991096 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:44287 every 8 connection(s)
I20260929 11:33:07.992475 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:07.996147 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:07.998389 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8572
I20260929 11:33:07.998616 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:07.999377 6183 external_mini_cluster.cc:1428] Setting key 4baf4e0b430c87e80caea8577eba91eb
I20260929 11:33:08.008642 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e: Bootstrap starting.
I20260929 11:33:08.012475 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:08.013854 8639 log.cc:824] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:33:08 dist-test-slave-c0l9 krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:08.016839 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e: No bootstrap required, opened a new log
I20260929 11:33:08.022831 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:08.001510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.6.9.254:44287)
0929 11:33:08.002417 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:08.002433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:08.002476 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:33:08.004155 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:08.004178 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:08.004565 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:08.004966 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:08.006729 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.008101 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.009886 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.010605 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.010765 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:08.015602 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:08.015612 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:08.015629 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:08.015669 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:08.018709 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:08.019434 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:08.019441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:08.019447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:08.019620 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:08.019992 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:08.019996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:08.019998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:08.020543 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:08.020894 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:08.021297 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 11:33:08.021793 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":349,"threads_started":1}
I20260929 11:33:08.024402 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } }
I20260929 11:33:08.024760 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:08.024855 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b54f7f8c0f4b788a9e5821d8dc7e1e, State: Initialized, Role: FOLLOWER
I20260929 11:33:08.025374 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } }
I20260929 11:33:08.025595 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:08.025717 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:08.025856 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:08.027875 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } }
I20260929 11:33:08.028324 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b54f7f8c0f4b788a9e5821d8dc7e1e; no voters:
I20260929 11:33:08.028906 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:08.029017 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:08.029589 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [term 1 LEADER]: Becoming Leader. State: Replica: 31b54f7f8c0f4b788a9e5821d8dc7e1e, State: Running, Role: LEADER
I20260929 11:33:08.030118 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } }
I20260929 11:33:08.031044 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:08.033095 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } } }
I20260929 11:33:08.033111 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b54f7f8c0f4b788a9e5821d8dc7e1e. Latest consensus state: current_term: 1 leader_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b54f7f8c0f4b788a9e5821d8dc7e1e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44287 } } }
I20260929 11:33:08.033574 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:08.033596 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:08.034495 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:08.040584 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:08.050414 8651 catalog_manager.cc:1347] Generated new cluster ID: 62a240c892d948dfbe34d07965cd9af2
I20260929 11:33:08.050539 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:08.074476 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:08.076450 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:08.089526 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b54f7f8c0f4b788a9e5821d8dc7e1e: Generated new TSK 0
I20260929 11:33:08.090664 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:08.132400 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:08.441159 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:08.441512 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:08.441664 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:08.449061 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:08.449177 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:08.449331 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:08.459337 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:08.460718 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:08.462083 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:08.475250 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:08.475323 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:08.475345 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:08.475641 8665 server_base.cc:1034] running on GCE node
I20260929 11:33:08.476512 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:08.477525 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:08.478734 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790681588478683 us; error 38 us; skew 500 ppm
I20260929 11:33:08.479068 8665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:08 dist-test-slave-c0l9 krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:08.482121 8665 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:08.483584 8665 webserver.cc:460] Webserver started at http://127.6.9.193:39965/ using document root <none> and password file <none>
I20260929 11:33:08.484277 8665 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:08.484386 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:08.484722 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:08.487670 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "50c0dbe3b3ae4ad4ba96185cdc10ecda"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "1d7c9acd25fa09fa2713d64bcca92847"
server_key_iv: "890139b06295211fbb58cce7c3b6312d"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.488444 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "50c0dbe3b3ae4ad4ba96185cdc10ecda"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "1d7c9acd25fa09fa2713d64bcca92847"
server_key_iv: "890139b06295211fbb58cce7c3b6312d"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.493626 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:33:08.497069 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:08.498363 8665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:33:08.498512 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "50c0dbe3b3ae4ad4ba96185cdc10ecda"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "1d7c9acd25fa09fa2713d64bcca92847"
server_key_iv: "890139b06295211fbb58cce7c3b6312d"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.498692 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:08.513137 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:08.517069 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:08.517325 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:08.518208 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:08.519486 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:08.519556 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:08.519666 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:08.519719 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:08.549563 8665 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:39675
I20260929 11:33:08.549609 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:39675 every 8 connection(s)
I20260929 11:33:08.550773 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:08.559645 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8665
I20260929 11:33:08.559852 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:08.560480 6183 external_mini_cluster.cc:1428] Setting key 3756b0e70fd023d00d39fc61e683026d
Sep 29 11:33:08 dist-test-slave-c0l9 krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:08.569772 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:08.553179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:59445 (local address 127.6.9.254:44287)
0929 11:33:08.553583 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:08.553588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:08.554507 (+ 919us) server_negotiation.cc:407] Connection header received
0929 11:33:08.556211 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:08.556216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:08.556291 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:08.556387 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:08.557816 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.558612 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.560178 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.560619 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.560757 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:08.564255 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:08.564261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:08.564264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:08.564297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:08.566683 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:08.567492 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:08.567497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:08.567499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:08.567621 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:08.568068 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:08.568071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:08.568073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:08.568403 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:08.568537 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:08.569200 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 11:33:08.569436 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":204,"threads_started":1}
I20260929 11:33:08.570554 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:08.553692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44287 (local address 127.6.9.193:59445)
0929 11:33:08.554361 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:08.554396 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:08.555950 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:08.556574 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:08.556582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:08.557124 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:08.557645 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:08.557661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.558739 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.558742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:08.560045 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:08.560057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:08.560859 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:08.560867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:08.561166 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:08.561840 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:08.561862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:08.564045 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:08.566811 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:08.566820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:08.566837 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:08.567373 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:08.567719 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:08.567723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:08.567726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:08.567976 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:08.568512 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:08.568517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:08.568878 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 11:33:08.569103 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 11:33:08.569481 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":253,"threads_started":1}
I20260929 11:33:08.571933 8793 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44287
I20260929 11:33:08.572238 8793 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:08.573069 8793 heartbeater.cc:511] Master 127.6.9.254:44287 requested a full tablet report, sending...
I20260929 11:33:08.575170 8604 ts_manager.cc:194] Registered new tserver with Master: 50c0dbe3b3ae4ad4ba96185cdc10ecda (127.6.9.193:39675)
I20260929 11:33:08.576589 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:59445
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:08.599886 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:08.910817 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:08.911091 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:08.911211 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:08.918481 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:08.918609 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:08.918757 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:08.928787 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:08.930107 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:08.931455 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:08.950479 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:08.950534 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:08.950547 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:08.951363 8799 server_base.cc:1034] running on GCE node
I20260929 11:33:08.951990 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:08.952982 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:08.954208 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790681588954149 us; error 50 us; skew 500 ppm
I20260929 11:33:08.954522 8799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:08 dist-test-slave-c0l9 krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:08.957726 8799 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:08.959275 8799 webserver.cc:460] Webserver started at http://127.6.9.194:37127/ using document root <none> and password file <none>
I20260929 11:33:08.959967 8799 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:08.960078 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:08.960379 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:08.963346 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "28ced90780824238afc17e2d7d656c65"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "64eaf2d333dfc92fa02ac2d5ef026b51"
server_key_iv: "96f7106291bce235e70f7e4180ffa160"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.964123 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "28ced90780824238afc17e2d7d656c65"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "64eaf2d333dfc92fa02ac2d5ef026b51"
server_key_iv: "96f7106291bce235e70f7e4180ffa160"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.969300 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 11:33:08.972715 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:08.974085 8799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:08.974236 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "28ced90780824238afc17e2d7d656c65"
format_stamp: "Formatted at 2026-09-29 11:33:08 on dist-test-slave-c0l9"
server_key: "64eaf2d333dfc92fa02ac2d5ef026b51"
server_key_iv: "96f7106291bce235e70f7e4180ffa160"
server_key_version: "encryptionkey@0"
I20260929 11:33:08.974434 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:08.993287 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:08.997248 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:08.997510 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:08.998417 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:08.999743 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:08.999817 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:08.999936 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:08.999994 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:09.029345 8799 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42673
I20260929 11:33:09.029415 8926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42673 every 8 connection(s)
I20260929 11:33:09.030563 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:09.036466 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8799
I20260929 11:33:09.036688 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:09.037226 6183 external_mini_cluster.cc:1428] Setting key 4ec0d8f919f5e3058a00e8ffc528417b
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:09.053097 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:09.033092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:38919 (local address 127.6.9.254:44287)
0929 11:33:09.033300 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:09.033305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:09.034387 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 11:33:09.036167 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:09.036171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:09.036242 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:09.036326 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:09.038244 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.039285 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.041662 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.042076 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.042192 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:09.046400 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:09.046406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:09.046411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:09.046451 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:09.049667 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:09.050522 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:09.050528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:09.050531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:09.050702 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:09.051154 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:09.051159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:09.051162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:09.051499 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:09.051647 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:09.052501 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 11:33:09.052774 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:33:09.054028 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:09.033514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44287 (local address 127.6.9.194:38919)
0929 11:33:09.034213 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:09.034251 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:09.035900 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:09.036601 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:09.036612 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:09.037349 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:09.038065 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:09.038089 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.039464 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.039470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:09.041517 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:09.041532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.042316 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.042322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:09.042613 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:09.043345 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:09.043388 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:09.046118 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:09.049871 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:09.049880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:09.049895 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:09.050388 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:09.050836 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:09.050840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:09.050843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:09.051043 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:09.051645 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:09.051651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:09.052011 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 11:33:09.052287 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 11:33:09.052874 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260929 11:33:09.055567 8927 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44287
I20260929 11:33:09.055846 8927 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:33:09.056535 8927 heartbeater.cc:511] Master 127.6.9.254:44287 requested a full tablet report, sending...
I20260929 11:33:09.058043 8604 ts_manager.cc:194] Registered new tserver with Master: 28ced90780824238afc17e2d7d656c65 (127.6.9.194:42673)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:33:09.059024 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:38919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:09.079859 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:09.383661 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:09.383945 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:09.384038 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:09.391201 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:09.391325 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:09.391434 8932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:09.401039 8932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:09.402290 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:09.403569 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:09.415951 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:09.416050 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:09.416020 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:09.416891 8932 server_base.cc:1034] running on GCE node
I20260929 11:33:09.417534 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:09.418547 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:09.419757 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790681589419706 us; error 40 us; skew 500 ppm
I20260929 11:33:09.420085 8932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:09.425829 8932 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:09.427428 8932 webserver.cc:460] Webserver started at http://127.6.9.195:39657/ using document root <none> and password file <none>
I20260929 11:33:09.428115 8932 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:09.428236 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:09.428542 8932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:09.431695 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "66e0262d14734ebd9664f243bbadc68b"
format_stamp: "Formatted at 2026-09-29 11:33:09 on dist-test-slave-c0l9"
server_key: "60bac89b32756654e025a3dd8e7b9652"
server_key_iv: "2d61007b57eae735d446756632506149"
server_key_version: "encryptionkey@0"
I20260929 11:33:09.432493 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "66e0262d14734ebd9664f243bbadc68b"
format_stamp: "Formatted at 2026-09-29 11:33:09 on dist-test-slave-c0l9"
server_key: "60bac89b32756654e025a3dd8e7b9652"
server_key_iv: "2d61007b57eae735d446756632506149"
server_key_version: "encryptionkey@0"
I20260929 11:33:09.439242 8932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 11:33:09.443352 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:09.444736 8932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:33:09.444891 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "66e0262d14734ebd9664f243bbadc68b"
format_stamp: "Formatted at 2026-09-29 11:33:09 on dist-test-slave-c0l9"
server_key: "60bac89b32756654e025a3dd8e7b9652"
server_key_iv: "2d61007b57eae735d446756632506149"
server_key_version: "encryptionkey@0"
I20260929 11:33:09.445060 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:09.466466 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:09.470541 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:09.470772 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:09.471791 8932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:09.473160 8932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:09.473234 8932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:09.473328 8932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:09.473373 8932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:09.502815 8932 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:40145
I20260929 11:33:09.502871 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:40145 every 8 connection(s)
I20260929 11:33:09.504117 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:09.505935 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 8932
I20260929 11:33:09.506106 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:09.506609 6183 external_mini_cluster.cc:1428] Setting key 4a90e2b1185f4c7eca0f89f7a451bc78
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:09.522300 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:09.507192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:49125 (local address 127.6.9.254:44287)
0929 11:33:09.507425 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:09.507433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:09.508380 (+ 947us) server_negotiation.cc:407] Connection header received
0929 11:33:09.510195 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:09.510200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:09.510267 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:09.510367 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:09.511667 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.512409 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.513857 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.514245 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.514343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:09.517367 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:09.517372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:09.517375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:09.517406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:09.519592 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:09.520219 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:09.520223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:09.520225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:09.520336 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:09.520727 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:09.520731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:09.520733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:09.520998 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:09.521128 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:09.521757 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 11:33:09.521979 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:33:09.523227 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:09.506953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44287 (local address 127.6.9.195:49125)
0929 11:33:09.508230 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:09.508273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:09.509948 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:09.510524 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:09.510531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:09.511006 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:09.511520 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:09.511540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.512520 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.512524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:09.513740 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:09.513751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:09.514401 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:09.514406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:09.514584 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:09.515163 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:09.515185 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:09.517172 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:09.519700 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:09.519706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:09.519716 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:09.520097 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:09.520435 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:09.520438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:09.520441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:09.520604 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:09.521117 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:09.521125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:09.521422 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 11:33:09.521644 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 11:33:09.522135 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":817,"threads_started":1}
I20260929 11:33:09.524513 9060 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44287
I20260929 11:33:09.524843 9060 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:09.525560 9060 heartbeater.cc:511] Master 127.6.9.254:44287 requested a full tablet report, sending...
I20260929 11:33:09.526947 8604 ts_manager.cc:194] Registered new tserver with Master: 66e0262d14734ebd9664f243bbadc68b (127.6.9.195:40145)
I20260929 11:33:09.527733 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:49125
I20260929 11:33:09.534058 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:33:09.560632 8795 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:09.558298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.6.9.254:44287)
0929 11:33:09.558479 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:09.558486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:09.558880 (+ 394us) server_negotiation.cc:407] Connection header received
0929 11:33:09.559063 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:09.559068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:09.559131 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:09.559214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:09.560399 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46720: BlockingRecv error: recv got EOF from 127.0.0.1:46720 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:33:09.564059 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8665
W20260929 11:33:09.566468 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8665: Not authorized: Client connection negotiation failed: client connection to 127.6.9.193:39675: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:33:09.566660 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8665
I20260929 11:33:09.579340 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8799
W20260929 11:33:09.581903 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8799: Not authorized: Client connection negotiation failed: client connection to 127.6.9.194:42673: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:33:09.582082 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8799
I20260929 11:33:09.593993 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8932
W20260929 11:33:09.596853 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8932: Not authorized: Client connection negotiation failed: client connection to 127.6.9.195:40145: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:33:09.596978 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8932
I20260929 11:33:09.609133 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 8572
I20260929 11:33:09.610358 8634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:09.712201 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 8572
2026-09-29T11:33:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3283 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[9077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[9077](info): set up 2 sockets
Sep 29 11:33:09 dist-test-slave-c0l9 krb5kdc[9077](info): commencing operation
krb5kdc: starting...
W20260929 11:33:11.009346 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:11 dist-test-slave-c0l9 krb5kdc[9077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:11.219707 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:37825
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:11.657452 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:11.657793 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:11.658082 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:11.671118 9090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:11.671342 9090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:11.671422 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:11.671494 9090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:11.671573 9090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:11.690240 9090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:11.692087 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:11.694268 9090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:11.709911 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:11.709941 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:11.710058 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:11.710553 9090 server_base.cc:1034] running on GCE node
I20260929 11:33:11.711963 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:11.714100 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:11.717420 9090 hybrid_clock.cc:648] HybridClock initialized: now 1790681591717339 us; error 113 us; skew 500 ppm
I20260929 11:33:11.717873 9090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:11 dist-test-slave-c0l9 krb5kdc[9077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681591, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:11.725847 9090 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:11.727662 9090 webserver.cc:460] Webserver started at http://127.6.9.254:35775/ using document root <none> and password file <none>
I20260929 11:33:11.728420 9090 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:11.728535 9090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:11.728962 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:11.734941 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e"
format_stamp: "Formatted at 2026-09-29 11:33:11 on dist-test-slave-c0l9"
server_key: "a0581afa76aceb201a7d7af6834619ac"
server_key_iv: "8f7571924cc30951a33950551101bb07"
server_key_version: "encryptionkey@0"
I20260929 11:33:11.736884 9090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e"
format_stamp: "Formatted at 2026-09-29 11:33:11 on dist-test-slave-c0l9"
server_key: "a0581afa76aceb201a7d7af6834619ac"
server_key_iv: "8f7571924cc30951a33950551101bb07"
server_key_version: "encryptionkey@0"
I20260929 11:33:11.746155 9090 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 11:33:11.752202 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:11.754196 9090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:33:11.754415 9090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e"
format_stamp: "Formatted at 2026-09-29 11:33:11 on dist-test-slave-c0l9"
server_key: "a0581afa76aceb201a7d7af6834619ac"
server_key_iv: "8f7571924cc30951a33950551101bb07"
server_key_version: "encryptionkey@0"
I20260929 11:33:11.754644 9090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:11.776901 9090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:11.783043 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:11.783509 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:11.809737 9090 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:37825
I20260929 11:33:11.809767 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:37825 every 8 connection(s)
I20260929 11:33:11.811374 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:11.816318 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9090
I20260929 11:33:11.816532 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:11.817137 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:11.817458 6183 external_mini_cluster.cc:1428] Setting key 8a7230d05c86c10a305750dca96c3386
Sep 29 11:33:11 dist-test-slave-c0l9 krb5kdc[9077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:11.840262 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e: Bootstrap starting.
I20260929 11:33:11.845628 9157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:11.847487 9157 log.cc:824] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:11.852326 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e: No bootstrap required, opened a new log
I20260929 11:33:11.852965 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:11.819942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.6.9.254:37825)
0929 11:33:11.820858 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:11.820872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:11.820945 (+ 73us) server_negotiation.cc:407] Connection header received
0929 11:33:11.823192 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:11.823219 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:11.823912 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:11.824255 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:11.827983 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:11.830313 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:11.833667 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:11.834919 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:11.835170 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:11.842472 (+ 7302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:11.842480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:11.842496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:11.842549 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:11.846313 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:11.847799 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:11.847835 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:11.847855 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:11.848223 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:11.848945 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:11.848950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:11.848953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:11.849902 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:11.850219 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:11.850815 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 11:33:11.851380 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":451,"threads_started":1}
I20260929 11:33:11.858081 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:33:11.865471 9157 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } }
I20260929 11:33:11.866096 9157 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:11.866465 9157 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f995cf7d614dff8e29e69a2cdfbe8e, State: Initialized, Role: FOLLOWER
I20260929 11:33:11.867520 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } }
I20260929 11:33:11.867818 9157 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:11.868098 9157 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:11.868286 9157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:11.871284 9157 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } }
I20260929 11:33:11.872092 9157 leader_election.cc:302] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f995cf7d614dff8e29e69a2cdfbe8e; no voters:
I20260929 11:33:11.872884 9157 leader_election.cc:288] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:11.873085 9163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:11.874274 9163 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [term 1 LEADER]: Becoming Leader. State: Replica: d9f995cf7d614dff8e29e69a2cdfbe8e, State: Running, Role: LEADER
I20260929 11:33:11.875219 9163 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } }
I20260929 11:33:11.875962 9157 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:11.880092 9164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } } }
I20260929 11:33:11.880594 9164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:11.881616 9165 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f995cf7d614dff8e29e69a2cdfbe8e. Latest consensus state: current_term: 1 leader_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f995cf7d614dff8e29e69a2cdfbe8e" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37825 } } }
I20260929 11:33:11.882099 9165 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:11.888177 9168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:11.897478 9168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:11.912701 9168 catalog_manager.cc:1347] Generated new cluster ID: a9785ee56f6e4b149935c615879da85a
I20260929 11:33:11.912828 9168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:11.929313 9168 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:11.932376 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:11.913439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.6.9.254:37825)
0929 11:33:11.913614 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:11.913620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:11.914621 (+ 1001us) server_negotiation.cc:407] Connection header received
0929 11:33:11.915243 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:11.915250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:11.915479 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:11.915759 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:11.917582 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:11.919547 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:11.923048 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:11.924156 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:11.924612 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:11.927869 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:11.927884 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:11.927896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:11.929577 (+ 1681us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:11.929827 (+ 250us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:11.929997 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:11.930253 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:11.930942 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 11:33:11.931406 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":68}
I20260929 11:33:11.933969 9168 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:11.949290 9168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f995cf7d614dff8e29e69a2cdfbe8e: Generated new TSK 0
I20260929 11:33:11.951644 9168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:33:11.967749 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9090
I20260929 11:33:11.969880 9152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:12.090461 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9090
2026-09-29T11:33:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2383 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:12 dist-test-slave-c0l9 krb5kdc[9192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:12 dist-test-slave-c0l9 krb5kdc[9192](info): set up 2 sockets
Sep 29 11:33:12 dist-test-slave-c0l9 krb5kdc[9192](info): commencing operation
krb5kdc: starting...
W20260929 11:33:13.605150 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.469s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:13 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:13.897601 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40167
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:14.333309 9205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:14.333642 9205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:14.333781 9205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:14.341109 9205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:14.341246 9205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:14.341325 9205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:14.341399 9205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:14.341459 9205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:14.351436 9205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:14.352859 9205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:14.354229 9205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:14.365749 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:14.365727 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:14.365688 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:14.366376 9205 server_base.cc:1034] running on GCE node
I20260929 11:33:14.367115 9205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:14.368501 9205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:14.369693 9205 hybrid_clock.cc:648] HybridClock initialized: now 1790681594369642 us; error 41 us; skew 500 ppm
I20260929 11:33:14.370020 9205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:14 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:14.373236 9205 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:14.374681 9205 webserver.cc:460] Webserver started at http://127.6.9.254:42303/ using document root <none> and password file <none>
I20260929 11:33:14.375347 9205 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:14.375459 9205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:14.375746 9205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:14.378744 9205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "b7f5983dfd384dbb8a70078662221089"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "eb02fe6e6d4f37d025b7678dbdf0ac5b"
server_key_iv: "12961a5f18f02eb65c3bc13ff7564331"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.379531 9205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "b7f5983dfd384dbb8a70078662221089"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "eb02fe6e6d4f37d025b7678dbdf0ac5b"
server_key_iv: "12961a5f18f02eb65c3bc13ff7564331"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.384759 9205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:33:14.388180 9220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:14.389576 9205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 11:33:14.389744 9205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "b7f5983dfd384dbb8a70078662221089"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "eb02fe6e6d4f37d025b7678dbdf0ac5b"
server_key_iv: "12961a5f18f02eb65c3bc13ff7564331"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.389926 9205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:14.415123 9205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:14.419176 9205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:14.419438 9205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:14.436291 9205 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40167
I20260929 11:33:14.436326 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40167 every 8 connection(s)
I20260929 11:33:14.437702 9205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:14.441375 9272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:14.445072 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9205
I20260929 11:33:14.445308 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:14.446147 6183 external_mini_cluster.cc:1428] Setting key c128d44447651dfa0f9d4da797da8671
I20260929 11:33:14.454993 9272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089: Bootstrap starting.
I20260929 11:33:14.458724 9272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:14.459700 9272 log.cc:824] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:14.462353 9272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089: No bootstrap required, opened a new log
Sep 29 11:33:14 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:14.470614 9272 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } }
I20260929 11:33:14.470973 9272 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:14.471072 9272 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f5983dfd384dbb8a70078662221089, State: Initialized, Role: FOLLOWER
I20260929 11:33:14.471628 9272 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } }
I20260929 11:33:14.471841 9272 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:14.471993 9272 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:14.472137 9272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:14.474893 9272 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } }
I20260929 11:33:14.475431 9272 leader_election.cc:302] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f5983dfd384dbb8a70078662221089; no voters:
I20260929 11:33:14.476012 9272 leader_election.cc:288] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:14.476171 9277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:14.476405 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:14.448410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.6.9.254:40167)
0929 11:33:14.449190 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:14.449203 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:14.449245 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:33:14.450839 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:14.450863 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:14.451309 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:14.451669 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:14.453497 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:14.454864 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:14.456561 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:14.457313 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:14.457445 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:14.469108 (+ 11663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:14.469125 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:14.469144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:14.469183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:14.472215 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:14.472998 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:14.473005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:14.473011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:14.473179 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:14.473577 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:14.473580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:14.473582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:14.474087 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:14.474321 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:14.474715 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 11:33:14.475064 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":362,"threads_started":1}
I20260929 11:33:14.477943 9272 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:14.478169 9277 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [term 1 LEADER]: Becoming Leader. State: Replica: b7f5983dfd384dbb8a70078662221089, State: Running, Role: LEADER
I20260929 11:33:14.478672 9277 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } }
I20260929 11:33:14.482249 9278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7f5983dfd384dbb8a70078662221089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } } }
I20260929 11:33:14.482580 9278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:14.482648 9279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f5983dfd384dbb8a70078662221089. Latest consensus state: current_term: 1 leader_uuid: "b7f5983dfd384dbb8a70078662221089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f5983dfd384dbb8a70078662221089" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40167 } } }
I20260929 11:33:14.483042 9279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:14.484579 9285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:14.489581 9285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:14.499047 9285 catalog_manager.cc:1347] Generated new cluster ID: 7eb8a18a28e44749943e7d0f0cf41b49
I20260929 11:33:14.499195 9285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:14.515406 9285 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:14.517406 9285 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:14.533281 9285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7f5983dfd384dbb8a70078662221089: Generated new TSK 0
I20260929 11:33:14.534164 9285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:14.583945 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:14.912375 9298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:14.912708 9298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:14.912858 9298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:14.920061 9298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:14.920176 9298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:14.920336 9298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:14.929934 9298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:14.931183 9298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:14.932417 9298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:14.945294 9304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:14.945358 9303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:14.945358 9306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:14.945948 9298 server_base.cc:1034] running on GCE node
I20260929 11:33:14.946593 9298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:14.947588 9298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:14.948770 9298 hybrid_clock.cc:648] HybridClock initialized: now 1790681594948714 us; error 45 us; skew 500 ppm
I20260929 11:33:14.949085 9298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:14 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:14.952229 9298 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:14.953660 9298 webserver.cc:460] Webserver started at http://127.6.9.193:34065/ using document root <none> and password file <none>
I20260929 11:33:14.954340 9298 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:14.954459 9298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:14.954738 9298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:14.957614 9298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "7c855743136a4d3fab31d3c752705943"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "e99f6e59942cd6cdfde0ad68c6d4ea47"
server_key_iv: "88d9eafdb073ff0268b366b430b2d2b4"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.958361 9298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "7c855743136a4d3fab31d3c752705943"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "e99f6e59942cd6cdfde0ad68c6d4ea47"
server_key_iv: "88d9eafdb073ff0268b366b430b2d2b4"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.963418 9298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:14.967020 9313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:14.968384 9298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:33:14.968575 9298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "7c855743136a4d3fab31d3c752705943"
format_stamp: "Formatted at 2026-09-29 11:33:14 on dist-test-slave-c0l9"
server_key: "e99f6e59942cd6cdfde0ad68c6d4ea47"
server_key_iv: "88d9eafdb073ff0268b366b430b2d2b4"
server_key_version: "encryptionkey@0"
I20260929 11:33:14.968844 9298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:14.982664 9298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:14.986953 9298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:14.987232 9298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:14.988200 9298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:14.989559 9298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:14.989635 9298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:14.989756 9298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:14.989817 9298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:15.019687 9298 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:39739
I20260929 11:33:15.019747 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:39739 every 8 connection(s)
I20260929 11:33:15.020959 9298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:15.023056 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9298
I20260929 11:33:15.023236 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:15.024009 6183 external_mini_cluster.cc:1428] Setting key c3b54473be06fce7d7ca8742ecfec06d
Sep 29 11:33:15 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681594, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:33:15.043354 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:15.023534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:56249 (local address 127.6.9.254:40167)
0929 11:33:15.024013 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:15.024020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:15.025034 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 11:33:15.027218 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:15.027223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:15.027289 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:15.027386 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:15.029361 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.030113 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.032148 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.032844 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.032992 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:15.036820 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:15.036828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:15.036833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:15.036872 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:15.040156 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:15.040980 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:15.040986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:15.040990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:15.041155 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:15.041609 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:15.041614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:15.041618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:15.041995 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:15.042163 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:15.042908 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 11:33:15.043100 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260929 11:33:15.044323 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:15.024045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40167 (local address 127.6.9.193:56249)
0929 11:33:15.024874 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:15.024923 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:15.026904 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:15.027650 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:15.027664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:15.028368 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:15.029166 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:15.029191 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.030266 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.030271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:15.031968 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:15.031981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.033087 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.033092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:15.033364 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:15.034116 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:15.034139 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:15.036556 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:15.040310 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:15.040317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:15.040331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:15.040846 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:15.041275 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:15.041278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:15.041281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:15.041489 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:15.042126 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:15.042132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:15.042518 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 11:33:15.042785 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 11:33:15.043192 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":287,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
I20260929 11:33:15.046087 9426 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40167
I20260929 11:33:15.046396 9426 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:15.047134 9426 heartbeater.cc:511] Master 127.6.9.254:40167 requested a full tablet report, sending...
I20260929 11:33:15.049306 9237 ts_manager.cc:194] Registered new tserver with Master: 7c855743136a4d3fab31d3c752705943 (127.6.9.193:39739)
I20260929 11:33:15.051062 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:56249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:15.066284 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:15.388392 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:15.388696 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:15.388815 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:15.396174 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:15.396294 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:15.396404 9432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:15.407796 9432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:15.409516 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:15.411172 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:15.426335 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:15.426358 9440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:15.426367 9438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:15.427059 9432 server_base.cc:1034] running on GCE node
I20260929 11:33:15.427796 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:15.429858 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:15.431069 9432 hybrid_clock.cc:648] HybridClock initialized: now 1790681595431033 us; error 63 us; skew 500 ppm
I20260929 11:33:15.431402 9432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:15 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:15.435581 9432 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:15.441816 9432 webserver.cc:460] Webserver started at http://127.6.9.194:43857/ using document root <none> and password file <none>
I20260929 11:33:15.443007 9432 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:15.443176 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:15.443737 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:15.447616 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "85df37234c5d47feaf0da817dde42a0b"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "af3e10b49a8fbc22b5278611236270ff"
server_key_iv: "7cb366d1fb01c0d1ab629313f43d14c2"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.448560 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "85df37234c5d47feaf0da817dde42a0b"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "af3e10b49a8fbc22b5278611236270ff"
server_key_iv: "7cb366d1fb01c0d1ab629313f43d14c2"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.454712 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 11:33:15.459020 9447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:15.460870 9432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:33:15.461093 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "85df37234c5d47feaf0da817dde42a0b"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "af3e10b49a8fbc22b5278611236270ff"
server_key_iv: "7cb366d1fb01c0d1ab629313f43d14c2"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.461355 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:15.517350 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:15.522742 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:15.523044 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:15.524055 9432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:15.525545 9432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:15.525633 9432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:15.525730 9432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:15.525774 9432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:15.555467 9432 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:39707
I20260929 11:33:15.555514 9559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:39707 every 8 connection(s)
I20260929 11:33:15.556772 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:15.565210 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9432
I20260929 11:33:15.565366 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:15.565783 6183 external_mini_cluster.cc:1428] Setting key 85143a9eb0a596089f0dac3b09485ad5
Sep 29 11:33:15 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:15.576112 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:15.559305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:54031 (local address 127.6.9.254:40167)
0929 11:33:15.559718 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:15.559722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:15.560705 (+ 983us) server_negotiation.cc:407] Connection header received
0929 11:33:15.562521 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:15.562525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:15.562594 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:15.562688 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:15.564077 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.564820 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.566383 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.566835 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.566967 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:15.570281 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:15.570287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:15.570292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:15.570328 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:15.573151 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:15.573882 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:15.573887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:15.573891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:15.574025 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:15.574471 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:15.574475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:15.574477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:15.574757 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:15.574875 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:15.575648 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 11:33:15.575818 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":200,"threads_started":1}
I20260929 11:33:15.577455 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:15.559846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40167 (local address 127.6.9.194:54031)
0929 11:33:15.560529 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:15.560575 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:15.562217 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:15.562896 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:15.562904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:15.563386 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:15.563939 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:15.563956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.564939 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.564942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:15.566243 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:15.566255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:15.567052 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:15.567059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:15.567288 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:15.567923 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:15.567949 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:15.570066 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:15.573288 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:15.573298 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:15.573315 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:15.573762 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:15.574129 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:15.574133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:15.574136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:15.574358 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:15.574875 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:15.574883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:15.575294 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 11:33:15.575589 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 11:33:15.576138 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":284,"threads_started":1}
I20260929 11:33:15.578886 9560 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40167
I20260929 11:33:15.579259 9560 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:15.580150 9560 heartbeater.cc:511] Master 127.6.9.254:40167 requested a full tablet report, sending...
I20260929 11:33:15.581889 9237 ts_manager.cc:194] Registered new tserver with Master: 85df37234c5d47feaf0da817dde42a0b (127.6.9.194:39707)
I20260929 11:33:15.583015 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:54031
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:15.607594 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:15.925529 9566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:15.925832 9566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:15.925966 9566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:15.933350 9566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:15.933473 9566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:15.933624 9566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:15.943838 9566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:15.945178 9566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:15.946447 9566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:15.959781 9571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:15.959780 9572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:15.959939 9574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:15.961050 9566 server_base.cc:1034] running on GCE node
I20260929 11:33:15.961733 9566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:15.962716 9566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:15.963923 9566 hybrid_clock.cc:648] HybridClock initialized: now 1790681595963867 us; error 54 us; skew 500 ppm
I20260929 11:33:15.964237 9566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:15 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:15.967398 9566 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:15.968974 9566 webserver.cc:460] Webserver started at http://127.6.9.195:46433/ using document root <none> and password file <none>
I20260929 11:33:15.969668 9566 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:15.969786 9566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:15.970072 9566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:15.973079 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "1c51f538e4104eaf8e9b4adfec0f688e"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "2e571ed8c635005def7b3e2bc53dd495"
server_key_iv: "ab09200ffdd9bc3fec1db28a284c2cb6"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.973867 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "1c51f538e4104eaf8e9b4adfec0f688e"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "2e571ed8c635005def7b3e2bc53dd495"
server_key_iv: "ab09200ffdd9bc3fec1db28a284c2cb6"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.979118 9566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 11:33:15.982625 9581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:15.983944 9566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:15.984093 9566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "1c51f538e4104eaf8e9b4adfec0f688e"
format_stamp: "Formatted at 2026-09-29 11:33:15 on dist-test-slave-c0l9"
server_key: "2e571ed8c635005def7b3e2bc53dd495"
server_key_iv: "ab09200ffdd9bc3fec1db28a284c2cb6"
server_key_version: "encryptionkey@0"
I20260929 11:33:15.984269 9566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:16.003576 9566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:16.007872 9566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:16.008162 9566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:16.009152 9566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:16.010551 9566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:16.010640 9566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:16.010763 9566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:16.010816 9566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:16.041985 9566 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:35281
I20260929 11:33:16.042032 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:35281 every 8 connection(s)
I20260929 11:33:16.043373 9566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:16.044394 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9566
I20260929 11:33:16.044562 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:16.045085 6183 external_mini_cluster.cc:1428] Setting key 047d34f2ec1f2a77c5511401ef17febf
I20260929 11:33:16.053903 9426 heartbeater.cc:503] Master 127.6.9.254:40167 was elected leader, sending a full tablet report...
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681595, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:16.063880 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.046039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54511 (local address 127.6.9.254:40167)
0929 11:33:16.046207 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:16.046212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:16.047645 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 11:33:16.049563 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:16.049567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:16.049658 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:16.049745 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:16.051189 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.051940 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.053611 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.054018 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.054137 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.057838 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:16.057843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:16.057846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:16.057882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:16.061140 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.061895 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.061899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.061902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.062030 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.062418 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.062422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.062424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.062724 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:16.062838 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:16.063464 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 11:33:16.063657 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:33:16.064975 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.046635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40167 (local address 127.6.9.195:54511)
0929 11:33:16.047483 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:16.047529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:16.049294 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:16.049940 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:16.049949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:16.050465 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:16.051021 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:16.051039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.052106 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.052110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:16.053456 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:16.053470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.054257 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.054266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:16.054491 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.055161 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:16.055189 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:16.057594 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:16.061281 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:16.061288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:16.061304 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:16.061759 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:16.062121 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:16.062124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:16.062127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:16.062313 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:16.062812 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:16.062818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:16.063170 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 11:33:16.063396 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 11:33:16.063802 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":322,"threads_started":1}
I20260929 11:33:16.066277 9694 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40167
I20260929 11:33:16.066556 9694 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:16.067308 9694 heartbeater.cc:511] Master 127.6.9.254:40167 requested a full tablet report, sending...
I20260929 11:33:16.068774 9237 ts_manager.cc:194] Registered new tserver with Master: 1c51f538e4104eaf8e9b4adfec0f688e (127.6.9.195:35281)
I20260929 11:33:16.069577 9237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:54511
I20260929 11:33:16.072964 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:16.119705 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.104977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.6.9.254:40167)
0929 11:33:16.105203 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:16.105211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:16.105593 (+ 382us) server_negotiation.cc:407] Connection header received
0929 11:33:16.105782 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:16.105787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:16.105867 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:16.105956 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:16.107624 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.108364 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.109826 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.110341 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.110468 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.114463 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:16.114468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:16.114471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:16.114514 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:16.116940 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.117652 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.117656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.117658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.117792 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.118171 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.118175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.118177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.118524 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:16.118718 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:16.119057 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 11:33:16.119349 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:33:16.120365 9237 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60204
I20260929 11:33:16.122252 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9298
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:16.139577 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.123227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.6.9.193:39739)
0929 11:33:16.123936 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:16.123942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:16.123966 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:33:16.124026 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:16.124033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:16.124242 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:16.124390 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:16.125875 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.127125 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.128610 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.129374 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.129493 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.133622 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:16.133637 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:16.133658 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:16.133703 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:16.136544 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.137337 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.137346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.137348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.137489 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.137885 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.137889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.137891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.138258 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:16.138547 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:16.138870 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:33:16.139164 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":383,"threads_started":1}
W20260929 11:33:16.141505 9421 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37518
W20260929 11:33:16.142982 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:16.143312 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9298
I20260929 11:33:16.158404 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9432
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:16.175891 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.159252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.6.9.194:39707)
0929 11:33:16.159957 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:16.159965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:16.159992 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:33:16.160064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:16.160070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:16.160284 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:16.160425 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:16.161698 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.162938 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.164464 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.165088 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.165197 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.169597 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:16.169610 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:16.169626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:16.169667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:16.172514 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.173155 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.173161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.173164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.173287 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.173700 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.173703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.173705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.174157 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:16.174466 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:16.175180 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 11:33:16.175397 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":381,"threads_started":1}
W20260929 11:33:16.177839 9555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52422
W20260929 11:33:16.179072 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9432: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:16.179260 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9432
I20260929 11:33:16.193892 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9566
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:16.210915 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:16.194927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.6.9.195:35281)
0929 11:33:16.195561 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:16.195567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:16.195597 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:33:16.195666 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:16.195673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:16.195890 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:16.196014 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:16.197518 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.198691 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.200140 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:16.200976 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:16.201074 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:16.205596 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:16.205608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:16.205623 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:16.205666 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:16.208162 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.208846 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.208853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.208856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.208979 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:16.209347 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:16.209351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:16.209353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:16.209700 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:16.209988 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:16.210314 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 11:33:16.210580 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
W20260929 11:33:16.212180 9689 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34266
W20260929 11:33:16.213392 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:33:16.213544 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9566
I20260929 11:33:16.225382 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9205
I20260929 11:33:16.226567 9267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:16.330386 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9205
2026-09-29T11:33:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4235 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9711](info): set up 2 sockets
Sep 29 11:33:16 dist-test-slave-c0l9 krb5kdc[9711](info): commencing operation
krb5kdc: starting...
W20260929 11:33:17.332985 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:17 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:17.519753 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:32803
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:17.831327 9724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:17.831666 9724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:17.831792 9724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:17.839104 9724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:17.839260 9724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:17.839344 9724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:17.839422 9724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:17.839483 9724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:17.849388 9724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:17.850713 9724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:17.852015 9724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:17.863570 9729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:17.863627 9730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:17.863731 9732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:17.864357 9724 server_base.cc:1034] running on GCE node
I20260929 11:33:17.865206 9724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:17.866779 9724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:17.867995 9724 hybrid_clock.cc:648] HybridClock initialized: now 1790681597867928 us; error 61 us; skew 500 ppm
I20260929 11:33:17.868340 9724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:17 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681597, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:17.871488 9724 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:17.873045 9724 webserver.cc:460] Webserver started at http://127.6.9.254:45441/ using document root <none> and password file <none>
I20260929 11:33:17.873736 9724 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:17.873853 9724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:17.874146 9724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:17.877141 9724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "8d79226d8486480a84059d9932845eaf"
format_stamp: "Formatted at 2026-09-29 11:33:17 on dist-test-slave-c0l9"
server_key: "f2b15220a23caa2b4a1050fe50de074f"
server_key_iv: "95c470537cfef38dfdea593a1c475063"
server_key_version: "encryptionkey@0"
I20260929 11:33:17.877882 9724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "8d79226d8486480a84059d9932845eaf"
format_stamp: "Formatted at 2026-09-29 11:33:17 on dist-test-slave-c0l9"
server_key: "f2b15220a23caa2b4a1050fe50de074f"
server_key_iv: "95c470537cfef38dfdea593a1c475063"
server_key_version: "encryptionkey@0"
I20260929 11:33:17.882994 9724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:33:17.886358 9739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:17.887689 9724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:33:17.887850 9724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "8d79226d8486480a84059d9932845eaf"
format_stamp: "Formatted at 2026-09-29 11:33:17 on dist-test-slave-c0l9"
server_key: "f2b15220a23caa2b4a1050fe50de074f"
server_key_iv: "95c470537cfef38dfdea593a1c475063"
server_key_version: "encryptionkey@0"
I20260929 11:33:17.888021 9724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:17.899837 9724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:17.903638 9724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:17.903887 9724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:17.920435 9724 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:32803
I20260929 11:33:17.920467 9790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:32803 every 8 connection(s)
I20260929 11:33:17.922130 9724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:17.925911 9791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:17.927155 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9724
I20260929 11:33:17.927397 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:17.928244 6183 external_mini_cluster.cc:1428] Setting key d89b780a88168001603a7ad47af42d65
I20260929 11:33:17.941023 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf: Bootstrap starting.
Sep 29 11:33:17 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:17.944998 9791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:17.946066 9791 log.cc:824] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:17.949357 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf: No bootstrap required, opened a new log
I20260929 11:33:17.952888 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:17.930469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.6.9.254:32803)
0929 11:33:17.931355 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:17.931371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:17.931417 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:33:17.933195 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:17.933220 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:17.933677 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:17.934053 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:17.935952 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:17.937364 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:17.939224 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:17.939954 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:17.940108 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:17.944843 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:17.944851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:17.944868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:17.944912 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:17.948300 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:17.949058 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:17.949067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:17.949075 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:17.949279 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:17.949646 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:17.949651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:17.949654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:17.950353 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:17.950661 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:17.951005 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 11:33:17.951463 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":435,"threads_started":1}
I20260929 11:33:17.958559 9791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } }
I20260929 11:33:17.958909 9791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:17.958982 9791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d79226d8486480a84059d9932845eaf, State: Initialized, Role: FOLLOWER
I20260929 11:33:17.959491 9791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } }
I20260929 11:33:17.959689 9791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:17.959779 9791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:17.959890 9791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:17.962024 9791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } }
I20260929 11:33:17.962713 9791 leader_election.cc:302] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d79226d8486480a84059d9932845eaf; no voters:
I20260929 11:33:17.963291 9791 leader_election.cc:288] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:17.963403 9796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:17.964020 9796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [term 1 LEADER]: Becoming Leader. State: Replica: 8d79226d8486480a84059d9932845eaf, State: Running, Role: LEADER
I20260929 11:33:17.964576 9796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } }
I20260929 11:33:17.965503 9791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:17.968600 9798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d79226d8486480a84059d9932845eaf. Latest consensus state: current_term: 1 leader_uuid: "8d79226d8486480a84059d9932845eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } } }
I20260929 11:33:17.968945 9798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:17.969769 9802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:17.969553 9797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d79226d8486480a84059d9932845eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d79226d8486480a84059d9932845eaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32803 } } }
I20260929 11:33:17.969916 9797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:17.974387 9802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:17.983104 9802 catalog_manager.cc:1347] Generated new cluster ID: b53861ebb2a94ef794d85e1fb74fa1f3
I20260929 11:33:17.983242 9802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:18.001420 9802 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:18.002887 9802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:18.009621 9802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d79226d8486480a84059d9932845eaf: Generated new TSK 0
I20260929 11:33:18.010421 9802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:18.053606 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:18.372219 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:18.372503 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:18.372628 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:18.379928 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:18.380043 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:18.380206 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:18.390506 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:18.391857 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:18.393224 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:18.405792 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:18.405856 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:18.405822 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:18.407035 9817 server_base.cc:1034] running on GCE node
I20260929 11:33:18.407646 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:18.408622 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:18.409888 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790681598409832 us; error 50 us; skew 500 ppm
I20260929 11:33:18.410210 9817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:18 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:18.413403 9817 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:18.414992 9817 webserver.cc:460] Webserver started at http://127.6.9.193:33571/ using document root <none> and password file <none>
I20260929 11:33:18.415693 9817 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:18.415828 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:18.416162 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:18.419351 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "7c70e77cf20942fe96b7e473c2da4bac"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "6435d8f1c398b2acae1093cef07151fd"
server_key_iv: "2ad45e79b7845fce32c97742a7dc5ed8"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.420279 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "7c70e77cf20942fe96b7e473c2da4bac"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "6435d8f1c398b2acae1093cef07151fd"
server_key_iv: "2ad45e79b7845fce32c97742a7dc5ed8"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.425961 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:33:18.429774 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.431190 9817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:33:18.431355 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "7c70e77cf20942fe96b7e473c2da4bac"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "6435d8f1c398b2acae1093cef07151fd"
server_key_iv: "2ad45e79b7845fce32c97742a7dc5ed8"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.431562 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:18.462746 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:18.467111 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:18.467366 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:18.468328 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:18.469590 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:18.469660 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.469767 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:18.469825 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.499261 9817 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:34595
I20260929 11:33:18.499296 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:34595 every 8 connection(s)
I20260929 11:33:18.500486 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:18.501430 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9817
I20260929 11:33:18.501648 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:18.502511 6183 external_mini_cluster.cc:1428] Setting key 4e1ff2dbe9b29886843ab9e4da5b7bd7
Sep 29 11:33:18 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:33:18.520633 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:18.503329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:43441 (local address 127.6.9.254:32803)
0929 11:33:18.504306 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:18.504313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:18.506760 (+ 2447us) server_negotiation.cc:407] Connection header received
0929 11:33:18.506812 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:18.506815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:18.506882 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:18.506960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:18.508892 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.509643 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.511630 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.512048 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.512155 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:18.515352 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:18.515357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:18.515361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:18.515396 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:18.517758 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:18.518414 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:18.518418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:18.518420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:18.518542 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:18.518931 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:18.518935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:18.518937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:18.519237 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:18.519365 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:18.520065 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 11:33:18.520287 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":213,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
I20260929 11:33:18.521643 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:18.503823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32803 (local address 127.6.9.193:43441)
0929 11:33:18.504588 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:18.504632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:18.506499 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:18.507241 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:18.507253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:18.507939 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:18.508693 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:18.508720 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.509814 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.509819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:18.511481 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:18.511493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.512252 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.512259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:18.512481 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:18.513121 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:18.513144 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:18.515147 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:18.517882 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:18.517891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:18.517907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:18.518295 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:18.518640 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:18.518645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:18.518648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:18.518824 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:18.519352 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:18.519359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:18.519677 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 11:33:18.519929 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 11:33:18.520421 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":327,"threads_started":1}
I20260929 11:33:18.523057 9945 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32803
I20260929 11:33:18.523344 9945 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:18.524030 9945 heartbeater.cc:511] Master 127.6.9.254:32803 requested a full tablet report, sending...
I20260929 11:33:18.526232 9756 ts_manager.cc:194] Registered new tserver with Master: 7c70e77cf20942fe96b7e473c2da4bac (127.6.9.193:34595)
I20260929 11:33:18.527770 9756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:43441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:18.543128 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:18.853029 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:18.853330 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:18.853457 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:18.860911 9951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:18.861058 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:18.861233 9951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:18.871361 9951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:18.872695 9951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:18.873992 9951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:18.886889 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:18.886950 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:18.886979 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:18.887709 9951 server_base.cc:1034] running on GCE node
I20260929 11:33:18.888298 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:18.889309 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:18.890504 9951 hybrid_clock.cc:648] HybridClock initialized: now 1790681598890450 us; error 50 us; skew 500 ppm
I20260929 11:33:18.890805 9951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:18 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:18.893810 9951 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:18.895205 9951 webserver.cc:460] Webserver started at http://127.6.9.194:36861/ using document root <none> and password file <none>
I20260929 11:33:18.895850 9951 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:18.895963 9951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:18.896246 9951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:18.899116 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "00f682c700ee42b3bc319ec2893fb91b"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "0ddf1b9fc3ec37f3c20b67dd4a7a9369"
server_key_iv: "cec6e1837e564715be8c3d52c64dfe74"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.899883 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "00f682c700ee42b3bc319ec2893fb91b"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "0ddf1b9fc3ec37f3c20b67dd4a7a9369"
server_key_iv: "cec6e1837e564715be8c3d52c64dfe74"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.904961 9951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 11:33:18.908365 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.909724 9951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:18.909883 9951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "00f682c700ee42b3bc319ec2893fb91b"
format_stamp: "Formatted at 2026-09-29 11:33:18 on dist-test-slave-c0l9"
server_key: "0ddf1b9fc3ec37f3c20b67dd4a7a9369"
server_key_iv: "cec6e1837e564715be8c3d52c64dfe74"
server_key_version: "encryptionkey@0"
I20260929 11:33:18.910073 9951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:18.928560 9951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:18.932956 9951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:18.933243 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:18.934180 9951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:18.935643 9951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:18.935724 9951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.935853 9951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:18.935916 9951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:18.966480 9951 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42247
I20260929 11:33:18.966547 10078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42247 every 8 connection(s)
I20260929 11:33:18.967715 9951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:18.970367 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 9951
I20260929 11:33:18.970566 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:18.970999 6183 external_mini_cluster.cc:1428] Setting key 27f531b5e9c61dd9e8214df76050b943
Sep 29 11:33:18 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681598, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:18.987779 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:18.970181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:43385 (local address 127.6.9.254:32803)
0929 11:33:18.970396 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:18.970403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:18.971624 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 11:33:18.973625 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:18.973630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:18.973721 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:18.973802 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:18.975229 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.976003 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.977632 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.978028 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.978131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:18.982029 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:18.982034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:18.982038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:18.982073 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:18.984552 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:18.985484 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:18.985488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:18.985491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:18.985618 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:18.986070 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:18.986073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:18.986075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:18.986354 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:18.986461 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:18.987333 (+ 872us) server_negotiation.cc:299] Negotiation successful
0929 11:33:18.987509 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:33:18.989135 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:18.970686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32803 (local address 127.6.9.194:43385)
0929 11:33:18.971435 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:18.971495 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:18.973369 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:18.974028 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:18.974036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:18.974539 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:18.975075 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:18.975092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.976146 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.976149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:18.977502 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:18.977514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:18.978233 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:18.978241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:18.978519 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:18.979136 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:18.979168 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:18.981786 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:18.984721 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:18.984731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:18.984751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:18.985355 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:18.985720 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:18.985725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:18.985729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:18.985973 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:18.986475 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:18.986493 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:18.986961 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 11:33:18.987278 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 11:33:18.987858 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":291,"threads_started":1}
I20260929 11:33:18.990465 10079 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:33:18.990801 10079 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:18.991706 10079 heartbeater.cc:511] Master 127.6.9.254:32803 requested a full tablet report, sending...
I20260929 11:33:18.993436 9756 ts_manager.cc:194] Registered new tserver with Master: 00f682c700ee42b3bc319ec2893fb91b (127.6.9.194:42247)
I20260929 11:33:18.994252 9756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:43385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:19.014175 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:19.330494 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:19.330813 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:19.330977 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:19.338404 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:19.338555 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:19.338725 10084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:19.349082 10084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:19.350808 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:19.352169 10084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:19.364691 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:19.364717 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:19.364826 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:19.365576 10084 server_base.cc:1034] running on GCE node
I20260929 11:33:19.366146 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:19.367122 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:19.368348 10084 hybrid_clock.cc:648] HybridClock initialized: now 1790681599368305 us; error 35 us; skew 500 ppm
I20260929 11:33:19.368744 10084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:19.371990 10084 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:19.373561 10084 webserver.cc:460] Webserver started at http://127.6.9.195:39859/ using document root <none> and password file <none>
I20260929 11:33:19.374219 10084 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:19.374327 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:19.374619 10084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:19.377588 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "6d5473a9d246406ea21034a64e0209df"
format_stamp: "Formatted at 2026-09-29 11:33:19 on dist-test-slave-c0l9"
server_key: "e5a986c8d67ca19b75376d23168581c9"
server_key_iv: "1e390a2f57835d36441639a5f63ec824"
server_key_version: "encryptionkey@0"
I20260929 11:33:19.378329 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5473a9d246406ea21034a64e0209df"
format_stamp: "Formatted at 2026-09-29 11:33:19 on dist-test-slave-c0l9"
server_key: "e5a986c8d67ca19b75376d23168581c9"
server_key_iv: "1e390a2f57835d36441639a5f63ec824"
server_key_version: "encryptionkey@0"
I20260929 11:33:19.383435 10084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:19.386859 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:19.388132 10084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:19.388273 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "6d5473a9d246406ea21034a64e0209df"
format_stamp: "Formatted at 2026-09-29 11:33:19 on dist-test-slave-c0l9"
server_key: "e5a986c8d67ca19b75376d23168581c9"
server_key_iv: "1e390a2f57835d36441639a5f63ec824"
server_key_version: "encryptionkey@0"
I20260929 11:33:19.388448 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:19.400315 10084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:19.404158 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:19.404403 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:19.405328 10084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:19.406534 10084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:19.406602 10084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:19.406704 10084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:19.406754 10084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:19.439827 10084 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:33583
I20260929 11:33:19.439865 10211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:33583 every 8 connection(s)
I20260929 11:33:19.441742 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:19.452126 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10084
I20260929 11:33:19.452329 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:19.452883 6183 external_mini_cluster.cc:1428] Setting key cf83ace2fc568bb15f1d47093cafabe3
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:19.466661 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.445449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:55203 (local address 127.6.9.254:32803)
0929 11:33:19.445660 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.445667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.447253 (+ 1586us) server_negotiation.cc:407] Connection header received
0929 11:33:19.450270 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.450277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.450400 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.450514 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.452811 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.453867 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.456446 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.456930 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.457029 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.461452 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.461459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.461464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.461504 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.463835 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.464491 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.464495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.464497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.464636 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.465072 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.465076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.465078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.465355 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.465468 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.466185 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.466378 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:33:19.467684 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.446071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32803 (local address 127.6.9.195:55203)
0929 11:33:19.447077 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:19.447124 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:19.449898 (+ 2774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:19.450881 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:19.450895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:19.451689 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:19.452601 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:19.452624 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.454084 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.454090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:19.456281 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:19.456306 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.457158 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.457166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:19.457463 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.458219 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:19.458250 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:19.461220 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:19.463961 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:19.463968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:19.463982 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:19.464399 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:19.464747 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:19.464751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:19.464755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:19.464938 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:19.465448 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:19.465454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:19.465839 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 11:33:19.466073 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 11:33:19.466470 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":306,"threads_started":1}
I20260929 11:33:19.469136 10212 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32803
I20260929 11:33:19.469434 10212 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:19.470091 10212 heartbeater.cc:511] Master 127.6.9.254:32803 requested a full tablet report, sending...
I20260929 11:33:19.471395 9755 ts_manager.cc:194] Registered new tserver with Master: 6d5473a9d246406ea21034a64e0209df (127.6.9.195:33583)
I20260929 11:33:19.472211 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:55203
I20260929 11:33:19.481086 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:19.529806 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.510316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.6.9.193:34595)
0929 11:33:19.510917 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.510924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.510949 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:33:19.511006 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.511013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.511218 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.511359 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.512718 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.513774 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.515025 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.515566 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.515682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.524332 (+ 8650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.524346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.524361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.524399 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.527092 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.527769 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.527776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.527779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.527907 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.528282 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.528286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.528288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.528637 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.528907 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.529228 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.529443 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":277,"threads_started":1}
I20260929 11:33:19.530536 9945 heartbeater.cc:503] Master 127.6.9.254:32803 was elected leader, sending a full tablet report...
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:19.546883 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.533903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.6.9.254:32803)
0929 11:33:19.534105 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.534110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.534275 (+ 165us) server_negotiation.cc:407] Connection header received
0929 11:33:19.534397 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.534401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.534460 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.534534 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.535682 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.536512 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.537801 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.538212 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.538306 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.542148 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.542156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.542161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.542204 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.544497 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.545192 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.545196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.545198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.545334 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.545691 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.545693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.545695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.546008 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.546181 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.546422 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.546579 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 11:33:19.547442 9755 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49246
I20260929 11:33:19.548415 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9817
I20260929 11:33:19.561486 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.549309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.6.9.193:34595)
0929 11:33:19.549465 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.549469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.549771 (+ 302us) server_negotiation.cc:407] Connection header received
0929 11:33:19.549954 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.549959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.550055 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.550149 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.551203 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.551882 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.553057 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.553465 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.553558 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.555640 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.555645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.555649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.555686 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.558637 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.559883 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.559888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.559890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.560032 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.560356 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.560360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.560362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.560624 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.560882 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.561099 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.561264 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:33:19.562078 9940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:19.643957 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9817
I20260929 11:33:19.657263 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9951
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:19.673071 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.658153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.6.9.194:42247)
0929 11:33:19.658677 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.658687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.658719 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:33:19.658800 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.658810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.659065 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.659233 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.660501 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.661548 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.662815 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.663339 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.663452 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.667538 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.667550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.667564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.667598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.670345 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.671008 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.671016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.671019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.671139 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.671484 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.671487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.671489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.671822 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.672110 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.672426 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.672718 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260929 11:33:19.674278 10074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:19.754403 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9951
I20260929 11:33:19.767289 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10084
Sep 29 11:33:19 dist-test-slave-c0l9 krb5kdc[9711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:19.783568 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:19.768304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.6.9.195:33583)
0929 11:33:19.768954 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:19.768962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:19.768991 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:33:19.769050 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:19.769056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:19.769283 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:19.769403 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:19.770719 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.771745 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.773046 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:19.773590 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:19.773686 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:19.778210 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:19.778222 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:19.778237 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:19.778273 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:19.781160 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.781797 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.781804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.781806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.781918 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:19.782225 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:19.782228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:19.782229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:19.782554 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:19.782777 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:19.783018 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 11:33:19.783220 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":338,"threads_started":1}
I20260929 11:33:19.784837 10207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:19.866762 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10084
I20260929 11:33:19.879169 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 9724
I20260929 11:33:19.880335 9786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:19.983167 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 9724
2026-09-29T11:33:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3652 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:20 dist-test-slave-c0l9 krb5kdc[10231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:20 dist-test-slave-c0l9 krb5kdc[10231](info): set up 2 sockets
Sep 29 11:33:20 dist-test-slave-c0l9 krb5kdc[10231](info): commencing operation
krb5kdc: starting...
W20260929 11:33:20.988765 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:21 dist-test-slave-c0l9 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:21.170128 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43113
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:21.480301 10244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:21.480603 10244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:21.480751 10244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:21.487911 10244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:21.488021 10244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:21.488068 10244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:21.488144 10244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:21.488211 10244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:21.498147 10244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:21.499462 10244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:21.500805 10244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:21.511698 10249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:21.511691 10250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:21.511960 10252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:21.512379 10244 server_base.cc:1034] running on GCE node
I20260929 11:33:21.513131 10244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:21.514420 10244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:21.515612 10244 hybrid_clock.cc:648] HybridClock initialized: now 1790681601515572 us; error 32 us; skew 500 ppm
I20260929 11:33:21.515945 10244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:21 dist-test-slave-c0l9 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:21.518812 10244 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:21.519904 10244 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:21.520006 10244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:21.520308 10244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:21.523185 10244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf"
format_stamp: "Formatted at 2026-09-29 11:33:21 on dist-test-slave-c0l9"
server_key: "82b388f03f393573b198b15eed4ba2f4"
server_key_iv: "cc0785d5d23d3dc48108b6598269eee3"
server_key_version: "encryptionkey@0"
I20260929 11:33:21.523940 10244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf"
format_stamp: "Formatted at 2026-09-29 11:33:21 on dist-test-slave-c0l9"
server_key: "82b388f03f393573b198b15eed4ba2f4"
server_key_iv: "cc0785d5d23d3dc48108b6598269eee3"
server_key_version: "encryptionkey@0"
I20260929 11:33:21.528810 10244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 11:33:21.532011 10258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:21.533265 10244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:21.533420 10244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf"
format_stamp: "Formatted at 2026-09-29 11:33:21 on dist-test-slave-c0l9"
server_key: "82b388f03f393573b198b15eed4ba2f4"
server_key_iv: "cc0785d5d23d3dc48108b6598269eee3"
server_key_version: "encryptionkey@0"
I20260929 11:33:21.533609 10244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:21.587623 10244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:21.591794 10244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:21.592047 10244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:21.608857 10244 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43113
I20260929 11:33:21.608835 10309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43113 every 8 connection(s)
I20260929 11:33:21.609719 10244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:21.613144 10310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:21.618180 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10244
I20260929 11:33:21.618400 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:21.619131 6183 external_mini_cluster.cc:1428] Setting key a899a2da15131f599bb29b74c76188de
I20260929 11:33:21.623946 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: Bootstrap starting.
I20260929 11:33:21.627329 10310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:21.628504 10310 log.cc:824] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:21.631704 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: No bootstrap required, opened a new log
Sep 29 11:33:21 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:21.640051 10310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } }
I20260929 11:33:21.640439 10310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:21.640537 10310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74373bbd349e4b5ca8cb1fe0ec6f22bf, State: Initialized, Role: FOLLOWER
I20260929 11:33:21.641253 10310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } }
I20260929 11:33:21.641527 10310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:21.641670 10310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:21.641842 10310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:21.642918 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:21.621203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.6.9.254:43113)
0929 11:33:21.622067 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:21.622078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:21.622124 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:33:21.623751 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:21.623773 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:21.624158 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:21.624454 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:21.626290 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:21.627600 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:21.629367 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:21.630205 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:21.630343 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:21.634940 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:21.634948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:21.634967 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:21.635008 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:21.638202 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:21.638982 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:21.638995 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:21.639003 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:21.639178 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:21.639556 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:21.639561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:21.639565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:21.640180 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:21.640464 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:21.640968 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 11:33:21.641409 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":405,"threads_started":1}
I20260929 11:33:21.645138 10310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } }
I20260929 11:33:21.645690 10310 leader_election.cc:302] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74373bbd349e4b5ca8cb1fe0ec6f22bf; no voters:
I20260929 11:33:21.646335 10310 leader_election.cc:288] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:21.646464 10315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:21.647087 10315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [term 1 LEADER]: Becoming Leader. State: Replica: 74373bbd349e4b5ca8cb1fe0ec6f22bf, State: Running, Role: LEADER
I20260929 11:33:21.647652 10315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } }
I20260929 11:33:21.648460 10310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:21.650779 10317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74373bbd349e4b5ca8cb1fe0ec6f22bf. Latest consensus state: current_term: 1 leader_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } } }
I20260929 11:33:21.651187 10317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:21.651381 10316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74373bbd349e4b5ca8cb1fe0ec6f22bf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43113 } } }
I20260929 11:33:21.651684 10316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:21.654429 10328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:21.658589 10328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:21.666783 10328 catalog_manager.cc:1347] Generated new cluster ID: bfad254ce3d7420187c9a8dac41c8e7f
I20260929 11:33:21.666913 10328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:21.696173 10328 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:21.698206 10328 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:21.708135 10328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: Generated new TSK 0
I20260929 11:33:21.709064 10328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:21.754613 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:22.070983 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:22.071300 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:22.071427 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:22.078675 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:22.078812 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:22.078970 10336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:22.088800 10336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:22.090147 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:22.091513 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:22.104415 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:22.104429 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:22.104907 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:22.105088 10336 server_base.cc:1034] running on GCE node
I20260929 11:33:22.105722 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:22.106730 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:22.107950 10336 hybrid_clock.cc:648] HybridClock initialized: now 1790681602107903 us; error 38 us; skew 500 ppm
I20260929 11:33:22.108276 10336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:22 dist-test-slave-c0l9 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681602, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:22.111366 10336 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:22.112488 10336 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:22.112582 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:22.112926 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:22.115867 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "66e997dfb9324b99ba11c259729741ff"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "0a3c4ac3d38cf2518012d6fcda40320e"
server_key_iv: "2438ee2210d57d5e52eee54de25d25de"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.116708 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "66e997dfb9324b99ba11c259729741ff"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "0a3c4ac3d38cf2518012d6fcda40320e"
server_key_iv: "2438ee2210d57d5e52eee54de25d25de"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.121903 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:33:22.125375 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.126696 10336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:22.126845 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "66e997dfb9324b99ba11c259729741ff"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "0a3c4ac3d38cf2518012d6fcda40320e"
server_key_iv: "2438ee2210d57d5e52eee54de25d25de"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.127044 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:22.145992 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:22.150272 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:22.150527 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:22.151600 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:22.153098 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:22.153189 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.153293 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:22.153368 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.184820 10336 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:45709
I20260929 11:33:22.184861 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:45709 every 8 connection(s)
I20260929 11:33:22.185626 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:22.194271 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10336
I20260929 11:33:22.194473 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:22.195051 6183 external_mini_cluster.cc:1428] Setting key 201660e9f9a6d87baa38fcd6f06a1824
Sep 29 11:33:22 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681602, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:22.204691 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:22.187902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58427 (local address 127.6.9.254:43113)
0929 11:33:22.188363 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:22.188369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:22.189303 (+ 934us) server_negotiation.cc:407] Connection header received
0929 11:33:22.191016 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:22.191021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:22.191088 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:22.191182 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:22.192803 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.193560 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.195188 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.195583 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.195700 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:22.199243 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:22.199250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:22.199266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:22.199300 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:22.201695 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:22.202415 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:22.202419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:22.202422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:22.202542 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:22.202949 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:22.202953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:22.202955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:22.203231 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:22.203365 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:22.204071 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 11:33:22.204303 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":213,"threads_started":1}
I20260929 11:33:22.205672 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:22.188447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43113 (local address 127.6.9.193:58427)
0929 11:33:22.189139 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:22.189183 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:22.190777 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:22.191372 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:22.191380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:22.192029 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:22.192600 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:22.192620 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.193691 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.193694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:22.195039 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:22.195052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.195809 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.195818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:22.196099 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:22.196876 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:22.196905 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:22.199027 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:22.201830 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:22.201839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:22.201859 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:22.202291 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:22.202644 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:22.202649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:22.202653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:22.202837 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:22.203361 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:22.203370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:22.203705 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 11:33:22.203945 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 11:33:22.204500 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":290,"threads_started":1}
I20260929 11:33:22.207376 10463 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43113
I20260929 11:33:22.207770 10463 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:22.208623 10463 heartbeater.cc:511] Master 127.6.9.254:43113 requested a full tablet report, sending...
I20260929 11:33:22.211284 10275 ts_manager.cc:194] Registered new tserver with Master: 66e997dfb9324b99ba11c259729741ff (127.6.9.193:45709)
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:33:22.213050 10275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:58427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:22.235023 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:22.550448 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:22.550737 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:22.550834 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:22.558126 10469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:22.558248 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:22.558367 10469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:22.568114 10469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:22.569427 10469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:22.570665 10469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:22.582675 10474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:22.582703 10477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:22.582650 10475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:22.583369 10469 server_base.cc:1034] running on GCE node
I20260929 11:33:22.583978 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:22.584923 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:22.586104 10469 hybrid_clock.cc:648] HybridClock initialized: now 1790681602586050 us; error 51 us; skew 500 ppm
I20260929 11:33:22.586406 10469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:22 dist-test-slave-c0l9 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681602, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:22.589388 10469 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:22.590476 10469 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:22.590567 10469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:22.590849 10469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:22.593802 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "70a81dad42948dba25cdff13c9f19ea0"
server_key_iv: "8ce5dac2ff2ee72f6a08f0e3f1dcb13a"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.594619 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "70a81dad42948dba25cdff13c9f19ea0"
server_key_iv: "8ce5dac2ff2ee72f6a08f0e3f1dcb13a"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.599725 10469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:22.603170 10483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.604570 10469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:22.604777 10469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda"
format_stamp: "Formatted at 2026-09-29 11:33:22 on dist-test-slave-c0l9"
server_key: "70a81dad42948dba25cdff13c9f19ea0"
server_key_iv: "8ce5dac2ff2ee72f6a08f0e3f1dcb13a"
server_key_version: "encryptionkey@0"
I20260929 11:33:22.604997 10469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:22.620900 10469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:22.624955 10469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:22.625205 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:22.626118 10469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:22.627421 10469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:22.627516 10469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.627619 10469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:22.627689 10469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:22.657976 10469 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:41109
I20260929 11:33:22.658026 10595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:41109 every 8 connection(s)
I20260929 11:33:22.658829 10469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:22.662323 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10469
I20260929 11:33:22.662500 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:22.662951 6183 external_mini_cluster.cc:1428] Setting key 5a82378768bea7900fe7d539e3dbb48a
Sep 29 11:33:22 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681602, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:33:22.678838 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:22.661384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:35417 (local address 127.6.9.254:43113)
0929 11:33:22.661554 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:22.661558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:22.662925 (+ 1367us) server_negotiation.cc:407] Connection header received
0929 11:33:22.665019 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:22.665026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:22.665115 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:22.665207 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:22.666829 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.667597 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.669328 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.669752 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.669868 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:22.673469 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:22.673475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:22.673478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:22.673510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:22.675915 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:22.676626 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:22.676630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:22.676632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:22.676800 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:22.677209 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:22.677213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:22.677215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:22.677475 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:22.677605 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:22.678294 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 11:33:22.678520 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:33:22.679958 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:22.661855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43113 (local address 127.6.9.194:35417)
0929 11:33:22.662719 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:22.662774 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:22.664732 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:22.665466 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:22.665475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:22.666035 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:22.666641 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:22.666667 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.667756 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.667760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:22.669164 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:22.669178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:22.669997 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:22.670006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:22.670230 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:22.670933 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:22.670956 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:22.673254 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:22.676058 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:22.676067 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:22.676085 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:22.676498 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:22.676902 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:22.676907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:22.676911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:22.677091 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:22.677602 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:22.677610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:22.677927 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 11:33:22.678171 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 11:33:22.678713 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:33:22.681559 10596 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43113
I20260929 11:33:22.681842 10596 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:22.682457 10596 heartbeater.cc:511] Master 127.6.9.254:43113 requested a full tablet report, sending...
I20260929 11:33:22.683825 10275 ts_manager.cc:194] Registered new tserver with Master: bbe12e8b75d34f9288a0a1c7adb3cbda (127.6.9.194:41109)
I20260929 11:33:22.684849 10275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:35417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:22.703166 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:23.016389 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:23.016772 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:23.016953 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:23.024858 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:23.024997 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:23.025166 10601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:23.035436 10601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:23.036865 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:23.038271 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:23.051214 10607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:23.051299 10606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:23.051331 10609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:23.052083 10601 server_base.cc:1034] running on GCE node
I20260929 11:33:23.052780 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:23.053771 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:23.054975 10601 hybrid_clock.cc:648] HybridClock initialized: now 1790681603054920 us; error 48 us; skew 500 ppm
I20260929 11:33:23.055305 10601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:23 dist-test-slave-c0l9 krb5kdc[10231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681603, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:23.058459 10601 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:23.059629 10601 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:23.059720 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:23.059983 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:23.062886 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "7ef4a1daee0640e4809cdcd1ed301ef3"
format_stamp: "Formatted at 2026-09-29 11:33:23 on dist-test-slave-c0l9"
server_key: "6a484d39b465ceb8549c78e67f7bf61e"
server_key_iv: "c18c41829a9bc19dffdb868084a09a35"
server_key_version: "encryptionkey@0"
I20260929 11:33:23.063668 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "7ef4a1daee0640e4809cdcd1ed301ef3"
format_stamp: "Formatted at 2026-09-29 11:33:23 on dist-test-slave-c0l9"
server_key: "6a484d39b465ceb8549c78e67f7bf61e"
server_key_iv: "c18c41829a9bc19dffdb868084a09a35"
server_key_version: "encryptionkey@0"
I20260929 11:33:23.068806 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 11:33:23.072383 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:23.073748 10601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:33:23.073917 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "7ef4a1daee0640e4809cdcd1ed301ef3"
format_stamp: "Formatted at 2026-09-29 11:33:23 on dist-test-slave-c0l9"
server_key: "6a484d39b465ceb8549c78e67f7bf61e"
server_key_iv: "c18c41829a9bc19dffdb868084a09a35"
server_key_version: "encryptionkey@0"
I20260929 11:33:23.074122 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:23.091729 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:23.095956 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:23.096213 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:23.097317 10601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:23.098847 10601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:23.098932 10601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:23.099040 10601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:23.099105 10601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:23.129040 10601 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:37323
I20260929 11:33:23.129099 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:37323 every 8 connection(s)
I20260929 11:33:23.129879 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:23.130980 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10601
I20260929 11:33:23.131174 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:23.131851 6183 external_mini_cluster.cc:1428] Setting key 406267139e4fe4927eb652cc5551dc34
Sep 29 11:33:23 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681603, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:23.151805 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.134153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:53813 (local address 127.6.9.254:43113)
0929 11:33:23.134313 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.134317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.135792 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 11:33:23.137705 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.137710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.137783 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.137888 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:23.139533 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.140299 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.141906 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.142336 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.142455 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.145939 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:23.145945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:23.145950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:23.145990 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:23.148970 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.149722 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.149726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.149729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.149855 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.150221 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.150224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.150226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.150504 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:23.150620 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.151298 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.151521 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:33:23.152748 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.134897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43113 (local address 127.6.9.195:53813)
0929 11:33:23.135643 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.135688 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.137411 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.138168 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.138178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.138735 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:23.139349 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.139370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.140428 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.140431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.141768 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.141782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.142588 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.142594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.142797 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.143458 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:23.143482 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:23.145705 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:23.149093 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:23.149101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:23.149114 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:23.149622 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:23.149950 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:23.149955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:23.149959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:23.150136 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:23.150597 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:23.150603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:23.150929 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.151159 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.151619 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":323,"threads_started":1}
I20260929 11:33:23.154072 10728 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43113
I20260929 11:33:23.154371 10728 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:23.155105 10728 heartbeater.cc:511] Master 127.6.9.254:43113 requested a full tablet report, sending...
I20260929 11:33:23.156476 10275 ts_manager.cc:194] Registered new tserver with Master: 7ef4a1daee0640e4809cdcd1ed301ef3 (127.6.9.195:37323)
I20260929 11:33:23.157316 10275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:53813
I20260929 11:33:23.159837 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:33:23.175918 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.164083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.6.9.254:43113)
0929 11:33:23.164257 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.164261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.164675 (+ 414us) server_negotiation.cc:407] Connection header received
0929 11:33:23.164818 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.164822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.164884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.164975 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:23.166286 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.167032 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.168342 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.168873 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.168974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.171123 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:23.171128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:23.171132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:23.171165 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:23.173592 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.174196 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.174201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.174203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.174344 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.174737 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.174741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.174743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.175057 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:23.175254 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.175526 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.175691 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:33:23.180214 10275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:23.188064 10275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:23.216147 10463 heartbeater.cc:503] Master 127.6.9.254:43113 was elected leader, sending a full tablet report...
I20260929 11:33:23.220271 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.199747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37323 (local address 127.0.0.1:33212)
0929 11:33:23.210571 (+ 10824us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.210610 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.210823 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.211681 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.211690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.211729 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:23.212096 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.212125 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.214604 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.214610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.217151 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.217168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.219176 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.219183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.219411 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.219472 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.219579 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.219791 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10562,"mutex_wait_us":1,"thread_start_us":10423,"threads_started":1}
I20260929 11:33:23.220908 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.199951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45709 (local address 127.0.0.1:50994)
0929 11:33:23.212460 (+ 12509us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.212477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.212595 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.213336 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.213341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.213365 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:23.213827 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.213842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.216032 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.216038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.218238 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.218255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.220120 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.220127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.220316 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.220348 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.220406 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.220527 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12391,"thread_start_us":12270,"threads_started":1}
I20260929 11:33:23.221645 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.200001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.6.9.195:37323)
0929 11:33:23.210189 (+ 10188us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.210201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.211026 (+ 825us) server_negotiation.cc:407] Connection header received
0929 11:33:23.211149 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.211157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.211422 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.212326 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:23.212630 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.214427 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.217406 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.218955 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.220827 (+ 1872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.221071 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.221152 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.221320 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9937,"thread_start_us":9739,"threads_started":1}
I20260929 11:33:23.221930 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.200506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.6.9.193:45709)
0929 11:33:23.212153 (+ 11647us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.212161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.212750 (+ 589us) server_negotiation.cc:407] Connection header received
0929 11:33:23.212874 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.212882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.213123 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.214038 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:23.214299 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.215881 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.218458 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.219895 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.221316 (+ 1421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.221485 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.221536 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.221646 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11499,"thread_start_us":11371,"threads_started":1}
I20260929 11:33:23.226217 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.213602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:41109 (local address 127.0.0.1:36876)
0929 11:33:23.218699 (+ 5097us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.218719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.218843 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.219616 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.219621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.219646 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:23.220015 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.220025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.222281 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.222286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.223778 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.223796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.225460 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.225466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.225631 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.225659 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.225749 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.225884 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4944,"thread_start_us":4788,"threads_started":1}
I20260929 11:33:23.226265 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.211348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.6.9.194:41109)
0929 11:33:23.218377 (+ 7029us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.218387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.218965 (+ 578us) server_negotiation.cc:407] Connection header received
0929 11:33:23.219084 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.219093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.219388 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.220258 (+ 870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:23.220471 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.222144 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.223985 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.225266 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.225381 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.225574 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.225806 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.225954 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6841,"thread_start_us":6694,"threads_started":1}
I20260929 11:33:23.225858 10398 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9556769f934cdb8bde44ea4871f98e (DEFAULT_TABLE table=test-table [id=83f72c67ec584cd091fce7d327e95baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:23.225859 10663 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9556769f934cdb8bde44ea4871f98e (DEFAULT_TABLE table=test-table [id=83f72c67ec584cd091fce7d327e95baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:23.227447 10663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9556769f934cdb8bde44ea4871f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:23.227447 10398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9556769f934cdb8bde44ea4871f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:23.229893 10531 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9556769f934cdb8bde44ea4871f98e (DEFAULT_TABLE table=test-table [id=83f72c67ec584cd091fce7d327e95baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:23.231087 10531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9556769f934cdb8bde44ea4871f98e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:23.244094 10748 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Bootstrap starting.
I20260929 11:33:23.248338 10748 tablet_bootstrap.cc:654] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:23.249478 10746 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Bootstrap starting.
I20260929 11:33:23.250195 10748 log.cc:824] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:23.250429 10747 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Bootstrap starting.
I20260929 11:33:23.253901 10748 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: No bootstrap required, opened a new log
I20260929 11:33:23.254173 10748 ts_tablet_manager.cc:1397] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:33:23.254451 10746 tablet_bootstrap.cc:654] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:23.255750 10746 log.cc:824] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:23.255864 10747 tablet_bootstrap.cc:654] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:23.256996 10747 log.cc:824] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:23.258797 10746 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: No bootstrap required, opened a new log
I20260929 11:33:23.259114 10746 ts_tablet_manager.cc:1397] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 11:33:23.261464 10747 tablet_bootstrap.cc:492] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: No bootstrap required, opened a new log
I20260929 11:33:23.263342 10747 ts_tablet_manager.cc:1397] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260929 11:33:23.263736 10748 raft_consensus.cc:348] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.264099 10748 raft_consensus.cc:374] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:23.264209 10748 raft_consensus.cc:729] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e997dfb9324b99ba11c259729741ff, State: Initialized, Role: FOLLOWER
I20260929 11:33:23.264858 10748 consensus_queue.cc:260] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.266633 10748 ts_tablet_manager.cc:1428] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260929 11:33:23.269420 10746 raft_consensus.cc:348] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.269850 10746 raft_consensus.cc:374] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:23.269951 10746 raft_consensus.cc:729] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe12e8b75d34f9288a0a1c7adb3cbda, State: Initialized, Role: FOLLOWER
I20260929 11:33:23.270563 10746 consensus_queue.cc:260] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.271667 10596 heartbeater.cc:503] Master 127.6.9.254:43113 was elected leader, sending a full tablet report...
I20260929 11:33:23.272282 10746 ts_tablet_manager.cc:1428] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260929 11:33:23.273283 10747 raft_consensus.cc:348] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.273612 10747 raft_consensus.cc:374] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:23.273710 10747 raft_consensus.cc:729] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef4a1daee0640e4809cdcd1ed301ef3, State: Initialized, Role: FOLLOWER
I20260929 11:33:23.274325 10747 consensus_queue.cc:260] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.275404 10728 heartbeater.cc:503] Master 127.6.9.254:43113 was elected leader, sending a full tablet report...
I20260929 11:33:23.277542 10747 ts_tablet_manager.cc:1428] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 11:33:23.313088 10752 raft_consensus.cc:482] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:23.313355 10752 raft_consensus.cc:504] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.315037 10752 leader_election.cc:288] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef4a1daee0640e4809cdcd1ed301ef3 (127.6.9.195:37323), bbe12e8b75d34f9288a0a1c7adb3cbda (127.6.9.194:41109)
I20260929 11:33:23.321868 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.315392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44161 (local address 127.6.9.195:37323)
0929 11:33:23.315612 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.315619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.316003 (+ 384us) server_negotiation.cc:407] Connection header received
0929 11:33:23.316194 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.316199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.316279 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.316371 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:23.317059 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.317958 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.319948 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.320857 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.320955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.321035 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.321462 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.321597 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:33:23.321966 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.315292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37323 (local address 127.6.9.193:44161)
0929 11:33:23.315898 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.315913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.316060 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.316503 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.316508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.316531 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:23.316898 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.316911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.318088 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.318094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.319791 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.319803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.321082 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.321089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.321291 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.321330 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.321424 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.321553 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":51,"thread_start_us":310,"threads_started":1}
I20260929 11:33:23.322059 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.316229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:34337 (local address 127.6.9.194:41109)
0929 11:33:23.316427 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.316433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.316714 (+ 281us) server_negotiation.cc:407] Connection header received
0929 11:33:23.316872 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.316879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.316952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.317036 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:23.317606 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.318416 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.320172 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.321157 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.321295 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.321386 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.321709 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.321842 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:33:23.322619 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.315984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:41109 (local address 127.6.9.193:34337)
0929 11:33:23.316601 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:23.316617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:23.316751 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:23.317059 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:23.317063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:23.317087 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:23.317444 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.317456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.318555 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.318558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.320007 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:23.320021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.321374 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.321379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:23.321549 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.321581 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:33:23.321669 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:33:23.321796 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":302,"threads_started":1}
I20260929 11:33:23.322983 10683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9556769f934cdb8bde44ea4871f98e" candidate_uuid: "66e997dfb9324b99ba11c259729741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" is_pre_election: true
I20260929 11:33:23.323570 10683 raft_consensus.cc:2454] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e997dfb9324b99ba11c259729741ff in term 0.
I20260929 11:33:23.323550 10551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9556769f934cdb8bde44ea4871f98e" candidate_uuid: "66e997dfb9324b99ba11c259729741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" is_pre_election: true
I20260929 11:33:23.324046 10551 raft_consensus.cc:2454] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e997dfb9324b99ba11c259729741ff in term 0.
I20260929 11:33:23.324306 10354 leader_election.cc:302] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e997dfb9324b99ba11c259729741ff, 7ef4a1daee0640e4809cdcd1ed301ef3; no voters:
I20260929 11:33:23.324764 10752 raft_consensus.cc:2781] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:23.324890 10752 raft_consensus.cc:482] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:23.324988 10752 raft_consensus.cc:3037] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:23.327132 10752 raft_consensus.cc:504] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.327683 10752 leader_election.cc:288] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [CANDIDATE]: Term 1 election: Requested vote from peers 7ef4a1daee0640e4809cdcd1ed301ef3 (127.6.9.195:37323), bbe12e8b75d34f9288a0a1c7adb3cbda (127.6.9.194:41109)
I20260929 11:33:23.328220 10551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9556769f934cdb8bde44ea4871f98e" candidate_uuid: "66e997dfb9324b99ba11c259729741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda"
I20260929 11:33:23.328202 10683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9556769f934cdb8bde44ea4871f98e" candidate_uuid: "66e997dfb9324b99ba11c259729741ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3"
I20260929 11:33:23.328418 10683 raft_consensus.cc:3037] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:23.328418 10551 raft_consensus.cc:3037] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:23.331614 10683 raft_consensus.cc:2454] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e997dfb9324b99ba11c259729741ff in term 1.
I20260929 11:33:23.331624 10551 raft_consensus.cc:2454] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e997dfb9324b99ba11c259729741ff in term 1.
I20260929 11:33:23.332052 10354 leader_election.cc:302] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e997dfb9324b99ba11c259729741ff, bbe12e8b75d34f9288a0a1c7adb3cbda; no voters:
I20260929 11:33:23.332356 10752 raft_consensus.cc:2781] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:23.332854 10752 raft_consensus.cc:686] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 1 LEADER]: Becoming Leader. State: Replica: 66e997dfb9324b99ba11c259729741ff, State: Running, Role: LEADER
I20260929 11:33:23.333330 10752 consensus_queue.cc:237] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } }
I20260929 11:33:23.337744 10274 catalog_manager.cc:5495] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e997dfb9324b99ba11c259729741ff (127.6.9.193). New cstate: current_term: 1 leader_uuid: "66e997dfb9324b99ba11c259729741ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e997dfb9324b99ba11c259729741ff" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:23.378621 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.373491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.6.9.193:45709)
0929 11:33:23.373646 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.373651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.373668 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:33:23.373789 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.373792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.373879 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.374001 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:23.374366 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.375065 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.376365 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.376796 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.376889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.377387 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:23.378008 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:23.378163 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.378270 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.378356 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 11:33:23.381263 10729 tablet.cc:2369] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:23.387746 10683 raft_consensus.cc:1261] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 1 FOLLOWER]: Refusing update from remote peer 66e997dfb9324b99ba11c259729741ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:23.387746 10551 raft_consensus.cc:1261] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 1 FOLLOWER]: Refusing update from remote peer 66e997dfb9324b99ba11c259729741ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:23.388902 10757 consensus_queue.cc:1035] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef4a1daee0640e4809cdcd1ed301ef3" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:23.389304 10752 consensus_queue.cc:1035] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe12e8b75d34f9288a0a1c7adb3cbda" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 41109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:23.403623 10761 mvcc.cc:205] Tried to move back new op lower bound from 7334631847463829504 to 7334631847256940544. Current Snapshot: MvccSnapshot[applied={T|T < 7334631847463829504}]
I20260929 11:33:23.415858 10274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:23.416133 10274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:23.419715 10274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74373bbd349e4b5ca8cb1fe0ec6f22bf: Sending DeleteTablet for 3 replicas of tablet 5d9556769f934cdb8bde44ea4871f98e
I20260929 11:33:23.420984 10398 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9556769f934cdb8bde44ea4871f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:23 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:50994
I20260929 11:33:23.421069 10531 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9556769f934cdb8bde44ea4871f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:23 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:36876
I20260929 11:33:23.421345 10663 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9556769f934cdb8bde44ea4871f98e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:23 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:33212
I20260929 11:33:23.422353 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10336
I20260929 11:33:23.431676 10771 tablet_replica.cc:330] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: stopping tablet replica
I20260929 11:33:23.432413 10771 raft_consensus.cc:2229] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:23.433678 10771 raft_consensus.cc:2258] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:23.436861 10769 tablet_replica.cc:330] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: stopping tablet replica
I20260929 11:33:23.440462 10769 raft_consensus.cc:2229] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:23.442915 10769 raft_consensus.cc:2258] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:23.443385 10771 ts_tablet_manager.cc:1905] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:23.446738 10770 tablet_replica.cc:330] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: stopping tablet replica
I20260929 11:33:23.447405 10770 raft_consensus.cc:2229] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:23.449788 10770 raft_consensus.cc:2258] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:23.449960 10769 ts_tablet_manager.cc:1905] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:23.452411 10771 ts_tablet_manager.cc:1918] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:23.452732 10771 log.cc:1196] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/5d9556769f934cdb8bde44ea4871f98e
I20260929 11:33:23.453384 10771 ts_tablet_manager.cc:1939] T 5d9556769f934cdb8bde44ea4871f98e P 7ef4a1daee0640e4809cdcd1ed301ef3: Deleting consensus metadata
I20260929 11:33:23.454197 10770 ts_tablet_manager.cc:1905] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:23.455875 10262 catalog_manager.cc:4841] TS 7ef4a1daee0640e4809cdcd1ed301ef3 (127.6.9.195:37323): tablet 5d9556769f934cdb8bde44ea4871f98e (table test-table [id=83f72c67ec584cd091fce7d327e95baa]) successfully deleted
Sep 29 11:33:23 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:23.457882 10769 ts_tablet_manager.cc:1918] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:23.458194 10769 log.cc:1196] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/5d9556769f934cdb8bde44ea4871f98e
I20260929 11:33:23.458840 10769 ts_tablet_manager.cc:1939] T 5d9556769f934cdb8bde44ea4871f98e P 66e997dfb9324b99ba11c259729741ff: Deleting consensus metadata
I20260929 11:33:23.461452 10259 catalog_manager.cc:4841] TS 66e997dfb9324b99ba11c259729741ff (127.6.9.193:45709): tablet 5d9556769f934cdb8bde44ea4871f98e (table test-table [id=83f72c67ec584cd091fce7d327e95baa]) successfully deleted
I20260929 11:33:23.463097 10770 ts_tablet_manager.cc:1918] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:23.463275 10770 log.cc:1196] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/5d9556769f934cdb8bde44ea4871f98e
I20260929 11:33:23.463847 10770 ts_tablet_manager.cc:1939] T 5d9556769f934cdb8bde44ea4871f98e P bbe12e8b75d34f9288a0a1c7adb3cbda: Deleting consensus metadata
I20260929 11:33:23.465714 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.424082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.6.9.193:45709)
0929 11:33:23.424306 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.424313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.443385 (+ 19072us) server_negotiation.cc:407] Connection header received
0929 11:33:23.443930 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.443936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.444019 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.444098 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:23.445861 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.447084 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.449022 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.449721 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.449835 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.458790 (+ 8955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:23.458803 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:23.458827 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:23.458884 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:23.462551 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.463334 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.463344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.463348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.463531 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.463924 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.463929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.463932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.464285 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:23.464573 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.464935 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.465173 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:33:23.465741 10262 catalog_manager.cc:4841] TS bbe12e8b75d34f9288a0a1c7adb3cbda (127.6.9.194:41109): tablet 5d9556769f934cdb8bde44ea4871f98e (table test-table [id=83f72c67ec584cd091fce7d327e95baa]) successfully deleted
I20260929 11:33:23.466415 10458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:23.564626 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10336
I20260929 11:33:23.580068 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10469
Sep 29 11:33:23 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:23.594146 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.580891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.6.9.194:41109)
0929 11:33:23.581086 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.581092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.581122 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:33:23.581184 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.581188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.581251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.581342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:23.582568 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.583320 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.584518 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.584936 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.585041 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.589041 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:23.589058 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:23.589087 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:23.589125 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:23.591638 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.592333 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.592341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.592343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.592493 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.592861 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.592864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.592867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.593272 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:23.593492 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.593698 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.593885 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:33:23.594859 10591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:23.683089 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10469
I20260929 11:33:23.696786 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10601
Sep 29 11:33:23 dist-test-slave-c0l9 krb5kdc[10231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:23.711894 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:23.697615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.6.9.195:37323)
0929 11:33:23.697802 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:23.697806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:23.697826 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:33:23.697975 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:23.697979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:23.698042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:23.698136 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:23.699328 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.700062 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.701384 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:23.701792 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:23.701888 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:23.706180 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:23.706200 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:23.706220 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:23.706273 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:23.709186 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.709837 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.709845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.709847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.710027 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:23.710403 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:23.710407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:23.710409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:23.710813 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:23.711042 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:23.711362 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 11:33:23.711577 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:33:23.712491 10723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:23.800357 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10601
I20260929 11:33:23.813437 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10244
I20260929 11:33:23.814605 10305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:23.921455 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10244
2026-09-29T11:33:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3938 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T11:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:23Z Disabled control of system clock
I20260929 11:33:23.979630 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43107
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:33:24.303217 10780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:24.303525 10780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:24.303634 10780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:24.310922 10780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:24.311038 10780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:24.311074 10780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:24.311128 10780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:24.311170 10780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:24.320502 10780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:24.321846 10780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:24.323602 10780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:24.334673 10786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:24.334728 10788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:24.334775 10785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:24.335417 10780 server_base.cc:1034] running on GCE node
I20260929 11:33:24.336167 10780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:24.337563 10780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:24.338784 10780 hybrid_clock.cc:648] HybridClock initialized: now 1790681604338737 us; error 41 us; skew 500 ppm
I20260929 11:33:24.339112 10780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:24.341260 10780 webserver.cc:460] Webserver started at http://127.6.9.254:33535/ using document root <none> and password file <none>
I20260929 11:33:24.341918 10780 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:24.342015 10780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:24.342317 10780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:24.345213 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "32a6d57c4cb34cd18531ee17f3fc2363"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "c29b8f4502e523d230ed5fc10236f823"
server_key_iv: "046b64c21197fb7e0451491f2b2f226d"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.345988 10780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "32a6d57c4cb34cd18531ee17f3fc2363"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "c29b8f4502e523d230ed5fc10236f823"
server_key_iv: "046b64c21197fb7e0451491f2b2f226d"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.351047 10780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:24.354415 10794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:24.355713 10780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:33:24.355862 10780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "32a6d57c4cb34cd18531ee17f3fc2363"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "c29b8f4502e523d230ed5fc10236f823"
server_key_iv: "046b64c21197fb7e0451491f2b2f226d"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.356057 10780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:24.361717 10780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:24.362465 10780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:24.362736 10780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:24.379334 10780 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43107
I20260929 11:33:24.379352 10845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43107 every 8 connection(s)
I20260929 11:33:24.380630 10780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:24.384207 10846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:24.386626 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10780
I20260929 11:33:24.386785 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:24.387368 6183 external_mini_cluster.cc:1428] Setting key e8b1a56f28cf09f81ac775eb281cd209
I20260929 11:33:24.395206 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:24.389309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.6.9.254:43107)
0929 11:33:24.390261 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:24.390272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:24.390312 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:33:24.391859 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:24.391879 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:24.392288 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:24.392654 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:24.393656 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:24.393662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:24.393675 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:24.393688 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:24.393753 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:24.393769 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:24.393958 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:24.394090 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 11:33:24.394338 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260929 11:33:24.395336 10846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: Bootstrap starting.
I20260929 11:33:24.398365 10846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:24.399441 10846 log.cc:824] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:24.402042 10846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: No bootstrap required, opened a new log
I20260929 11:33:24.409389 10846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } }
I20260929 11:33:24.409777 10846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:24.409870 10846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a6d57c4cb34cd18531ee17f3fc2363, State: Initialized, Role: FOLLOWER
I20260929 11:33:24.410418 10846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } }
I20260929 11:33:24.410638 10846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:24.410765 10846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:24.410909 10846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:24.413362 10846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } }
I20260929 11:33:24.413918 10846 leader_election.cc:302] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a6d57c4cb34cd18531ee17f3fc2363; no voters:
I20260929 11:33:24.414567 10846 leader_election.cc:288] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:24.414651 10851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:24.415416 10851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [term 1 LEADER]: Becoming Leader. State: Replica: 32a6d57c4cb34cd18531ee17f3fc2363, State: Running, Role: LEADER
I20260929 11:33:24.415972 10851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } }
I20260929 11:33:24.416721 10846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:24.418606 10853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a6d57c4cb34cd18531ee17f3fc2363. Latest consensus state: current_term: 1 leader_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } } }
I20260929 11:33:24.419000 10853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:24.418998 10852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a6d57c4cb34cd18531ee17f3fc2363" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43107 } } }
I20260929 11:33:24.419312 10852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:24.421003 10859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:24.426466 10859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:24.436082 10859 catalog_manager.cc:1347] Generated new cluster ID: 827873996a6e4dab84fe6ef2ba23bf04
I20260929 11:33:24.436203 10859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:24.444881 10859 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:24.445319 10859 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:24.459843 10859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: Generated new TSK 0
I20260929 11:33:24.461158 10859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:33:24.468163 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--builtin_ntp_servers=127.6.9.212:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:33:24.793433 10870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:24.793751 10870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:24.793867 10870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:24.801393 10870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:24.801514 10870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:24.801666 10870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:24.811206 10870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:24.812588 10870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:24.814448 10870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:24.827847 10875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:24.827914 10878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:24.827893 10876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:24.828778 10870 server_base.cc:1034] running on GCE node
I20260929 11:33:24.829384 10870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:24.830339 10870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:24.831595 10870 hybrid_clock.cc:648] HybridClock initialized: now 1790681604831522 us; error 42 us; skew 500 ppm
I20260929 11:33:24.831964 10870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:24.839464 10870 webserver.cc:460] Webserver started at http://127.6.9.193:37947/ using document root <none> and password file <none>
I20260929 11:33:24.840401 10870 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:24.840548 10870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:24.840942 10870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:24.845093 10870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "5f8107582abf4f07b77db2b18f7f0733"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "d51fcf18b606ca42f67e8c0f594349f3"
server_key_iv: "87da7be48df9c68b284832e9e8e2c99f"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.846398 10870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "5f8107582abf4f07b77db2b18f7f0733"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "d51fcf18b606ca42f67e8c0f594349f3"
server_key_iv: "87da7be48df9c68b284832e9e8e2c99f"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.855059 10870 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260929 11:33:24.860108 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:24.861999 10870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:33:24.862212 10870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "5f8107582abf4f07b77db2b18f7f0733"
format_stamp: "Formatted at 2026-09-29 11:33:24 on dist-test-slave-c0l9"
server_key: "d51fcf18b606ca42f67e8c0f594349f3"
server_key_iv: "87da7be48df9c68b284832e9e8e2c99f"
server_key_version: "encryptionkey@0"
I20260929 11:33:24.862488 10870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:24.869536 10870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:24.870479 10870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:24.870720 10870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:24.871833 10870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:24.873507 10870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:24.873625 10870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:24.873729 10870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:24.873785 10870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:24.905952 10870 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:42997
I20260929 11:33:24.906001 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:42997 every 8 connection(s)
I20260929 11:33:24.907301 10870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:24.908824 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 10870
I20260929 11:33:24.908998 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:24.909592 6183 external_mini_cluster.cc:1428] Setting key ff35e5329c2ce068dc54a625736963d9
I20260929 11:33:24.912437 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--builtin_ntp_servers=127.6.9.212:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 11:33:24.916055 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:24.909864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:48615 (local address 127.6.9.254:43107)
0929 11:33:24.910352 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:24.910358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:24.911602 (+ 1244us) server_negotiation.cc:407] Connection header received
0929 11:33:24.913602 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:24.913608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:24.913670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:24.913757 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:24.915349 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:24.915354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:24.915359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:24.915376 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:24.915422 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:24.915435 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:24.915546 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:24.915728 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 11:33:24.915784 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260929 11:33:24.917361 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:24.910463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43107 (local address 127.6.9.193:48615)
0929 11:33:24.911470 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:24.911506 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:24.913203 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:24.914085 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:24.914097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:24.914231 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:24.914986 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:24.914990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:24.915040 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:24.915154 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:24.915531 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:24.915539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:24.915547 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:33:24.915831 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 11:33:24.916113 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260929 11:33:24.919291 10997 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43107
I20260929 11:33:24.919656 10997 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:24.920392 10997 heartbeater.cc:511] Master 127.6.9.254:43107 requested a full tablet report, sending...
I20260929 11:33:24.922586 10811 ts_manager.cc:194] Registered new tserver with Master: 5f8107582abf4f07b77db2b18f7f0733 (127.6.9.193:42997)
W20260929 11:33:25.236385 11001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:25.236727 11001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:25.236872 11001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:25.244272 11001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:25.244400 11001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:25.244544 11001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:25.253937 11001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:25.255299 11001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:25.257144 11001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:25.271260 11006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:25.271183 11007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:25.271214 11009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:25.272086 11001 server_base.cc:1034] running on GCE node
I20260929 11:33:25.272751 11001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:25.273681 11001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:25.274914 11001 hybrid_clock.cc:648] HybridClock initialized: now 1790681605274849 us; error 47 us; skew 500 ppm
I20260929 11:33:25.275271 11001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:25.277628 11001 webserver.cc:460] Webserver started at http://127.6.9.194:46675/ using document root <none> and password file <none>
I20260929 11:33:25.278327 11001 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:25.278435 11001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:25.278752 11001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:25.281771 11001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "04b0963143104767802317d710dadda7"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "68c04a8b62871f4e6c019160afeb733e"
server_key_iv: "1344720965c0c80640bfd37a7f5406be"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.282585 11001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "04b0963143104767802317d710dadda7"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "68c04a8b62871f4e6c019160afeb733e"
server_key_iv: "1344720965c0c80640bfd37a7f5406be"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.287928 11001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:25.291579 11015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.292995 11001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:33:25.293169 11001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "04b0963143104767802317d710dadda7"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "68c04a8b62871f4e6c019160afeb733e"
server_key_iv: "1344720965c0c80640bfd37a7f5406be"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.293382 11001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:25.299050 11001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:25.299876 11001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:25.300135 11001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:25.301108 11001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:25.302529 11001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:25.302618 11001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.302721 11001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:25.302786 11001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.333001 11001 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:36713
I20260929 11:33:25.333089 11127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:36713 every 8 connection(s)
I20260929 11:33:25.334333 11001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:25.342299 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.336800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:35999 (local address 127.6.9.254:43107)
0929 11:33:25.337021 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.337027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.338446 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 11:33:25.340148 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.340153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.340225 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.340318 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.341621 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.341625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.341630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.341648 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.341683 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.341695 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.341794 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.341989 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.342051 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:33:25.343171 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.337381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43107 (local address 127.6.9.194:35999)
0929 11:33:25.338276 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:25.338324 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:25.339843 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:25.340528 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:25.340536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:25.340633 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:25.341317 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:25.341320 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:25.341359 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:25.341451 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:25.341772 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:25.341777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:25.341783 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 11:33:25.342003 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 11:33:25.342214 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260929 11:33:25.343634 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11001
I20260929 11:33:25.343803 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:25.344367 6183 external_mini_cluster.cc:1428] Setting key 42ea60a148ad3564462bbb4a85c15914
I20260929 11:33:25.344610 11128 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43107
I20260929 11:33:25.344964 11128 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:25.345589 11128 heartbeater.cc:511] Master 127.6.9.254:43107 requested a full tablet report, sending...
I20260929 11:33:25.347234 10811 ts_manager.cc:194] Registered new tserver with Master: 04b0963143104767802317d710dadda7 (127.6.9.194:36713)
I20260929 11:33:25.347352 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--builtin_ntp_servers=127.6.9.212:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:33:25.677626 11131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:25.677939 11131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:25.678054 11131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:25.685679 11131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:25.685801 11131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:25.685966 11131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:25.695250 11131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:25.696547 11131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:25.698369 11131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:25.710768 11137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:25.710687 11139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:25.710798 11136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:25.711902 11131 server_base.cc:1034] running on GCE node
I20260929 11:33:25.712527 11131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:25.713562 11131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:25.714779 11131 hybrid_clock.cc:648] HybridClock initialized: now 1790681605714726 us; error 49 us; skew 500 ppm
I20260929 11:33:25.715106 11131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:25.717494 11131 webserver.cc:460] Webserver started at http://127.6.9.195:45559/ using document root <none> and password file <none>
I20260929 11:33:25.718328 11131 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:25.718438 11131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:25.718746 11131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:25.721791 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "310917afa00c4dd3bce75c551d59cd43"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "1c5182be4cd0945b085f3d82b16aa234"
server_key_iv: "ee1a4852f0884259fdf5c69423982822"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.722700 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "310917afa00c4dd3bce75c551d59cd43"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "1c5182be4cd0945b085f3d82b16aa234"
server_key_iv: "ee1a4852f0884259fdf5c69423982822"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.727965 11131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 11:33:25.731503 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.732867 11131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:25.733021 11131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "310917afa00c4dd3bce75c551d59cd43"
format_stamp: "Formatted at 2026-09-29 11:33:25 on dist-test-slave-c0l9"
server_key: "1c5182be4cd0945b085f3d82b16aa234"
server_key_iv: "ee1a4852f0884259fdf5c69423982822"
server_key_version: "encryptionkey@0"
I20260929 11:33:25.733233 11131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:25.739105 11131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:25.739995 11131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:25.740255 11131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:25.741255 11131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:25.742718 11131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:25.742826 11131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.742937 11131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:25.743012 11131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:25.773815 11131 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:39991
I20260929 11:33:25.773859 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:39991 every 8 connection(s)
I20260929 11:33:25.775127 11131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:25.776943 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11131
I20260929 11:33:25.777101 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:25.777563 6183 external_mini_cluster.cc:1428] Setting key 367ba89466fabe71227517a89b40881e
I20260929 11:33:25.784180 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.779085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:59311 (local address 127.6.9.254:43107)
0929 11:33:25.779262 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.779267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.779909 (+ 642us) server_negotiation.cc:407] Connection header received
0929 11:33:25.781724 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.781729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.781800 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.781888 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.783397 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.783403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.783408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.783430 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.783463 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.783476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.783596 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.783813 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.783887 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:33:25.785228 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.778603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43107 (local address 127.6.9.195:59311)
0929 11:33:25.779754 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:25.779806 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:25.781443 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:25.782132 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:25.782141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:25.782242 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:25.783063 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:25.783066 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:25.783108 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:25.783203 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:25.783579 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:25.783591 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:25.783599 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:33:25.783887 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 11:33:25.784175 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":353,"threads_started":1}
I20260929 11:33:25.786868 11258 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43107
I20260929 11:33:25.787200 11258 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:25.787829 11258 heartbeater.cc:511] Master 127.6.9.254:43107 requested a full tablet report, sending...
I20260929 11:33:25.789449 10811 ts_manager.cc:194] Registered new tserver with Master: 310917afa00c4dd3bce75c551d59cd43 (127.6.9.195:39991)
I20260929 11:33:25.793397 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:33:25.800582 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.797842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.6.9.254:43107)
0929 11:33:25.798051 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.798058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.798598 (+ 540us) server_negotiation.cc:407] Connection header received
0929 11:33:25.798763 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.798767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.798840 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.798932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.799896 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.799901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.799904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.799919 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.799951 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.799962 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.800061 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.800236 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.800299 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:33:25.804826 10811 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:25.811555 10811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:25.840318 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.824621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.6.9.193:42997)
0929 11:33:25.835394 (+ 10773us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.835415 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.836618 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 11:33:25.836733 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.836741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.837009 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.837150 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.839368 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.839379 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.839396 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.839429 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.839493 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.839514 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.839664 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.839879 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.839980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10569,"thread_start_us":11711,"threads_started":1}
I20260929 11:33:25.840389 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.824496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42997 (local address 127.0.0.1:34260)
0929 11:33:25.836037 (+ 11541us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:25.836083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:25.836241 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:25.837599 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:25.837607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:25.837708 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:25.839038 (+ 1330us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:25.839042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:25.839074 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:25.839161 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:25.839672 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:25.839684 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:25.839692 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:33:25.839872 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 11:33:25.839977 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11256,"thread_start_us":14922,"threads_started":1}
I20260929 11:33:25.842921 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.825162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.6.9.195:39991)
0929 11:33:25.825964 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.825974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.839974 (+ 14000us) server_negotiation.cc:407] Connection header received
0929 11:33:25.840056 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.840065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.840306 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.841072 (+ 766us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.842142 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.842150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.842164 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.842186 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.842228 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.842242 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.842399 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.842469 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.842565 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":415,"threads_started":1}
I20260929 11:33:25.842957 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.824562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:39991 (local address 127.0.0.1:59898)
0929 11:33:25.839727 (+ 15165us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:25.839745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:25.839845 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:25.840803 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:25.840808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:25.840832 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:25.841518 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:25.841522 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:25.841541 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:25.841586 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:25.842346 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:25.842351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:25.842355 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 11:33:25.842482 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:33:25.842533 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15004,"thread_start_us":11033,"threads_started":1}
I20260929 11:33:25.845789 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.827633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36713 (local address 127.0.0.1:46138)
0929 11:33:25.840588 (+ 12955us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:25.840605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:25.840703 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:25.843579 (+ 2876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:25.843584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:25.843617 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:25.844136 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:25.844139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:25.844155 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:25.844198 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:25.845374 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:25.845379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:25.845384 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:33:25.845478 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:33:25.845529 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12870,"thread_start_us":16702,"threads_started":1}
I20260929 11:33:25.845867 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:25.828554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.6.9.194:36713)
0929 11:33:25.841899 (+ 13345us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:25.841907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:25.841934 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:33:25.842009 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:25.842017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:25.843389 (+ 1372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:25.843892 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:25.845191 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:25.845196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:25.845205 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:25.845217 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:25.845254 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:25.845270 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:25.845391 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:25.845489 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 11:33:25.845585 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13169,"thread_start_us":12762,"threads_started":1}
I20260929 11:33:25.845287 10932 tablet_service.cc:1467] Processing CreateTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 (DEFAULT_TABLE table=test-table [id=40a211cd133145da9a3d756e2b51dee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:25.846805 10932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f242392e4cd4fe5a96df22e6039e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:25.847266 11193 tablet_service.cc:1467] Processing CreateTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 (DEFAULT_TABLE table=test-table [id=40a211cd133145da9a3d756e2b51dee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:25.848687 11193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f242392e4cd4fe5a96df22e6039e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:25.848847 11063 tablet_service.cc:1467] Processing CreateTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 (DEFAULT_TABLE table=test-table [id=40a211cd133145da9a3d756e2b51dee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:25.850009 11063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f242392e4cd4fe5a96df22e6039e718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:25.860986 11276 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Bootstrap starting.
I20260929 11:33:25.864082 11276 tablet_bootstrap.cc:654] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:25.864995 11277 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Bootstrap starting.
I20260929 11:33:25.866952 11276 log.cc:824] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:25.867134 11278 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Bootstrap starting.
I20260929 11:33:25.869119 11277 tablet_bootstrap.cc:654] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:25.870150 11277 log.cc:824] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:25.871001 11278 tablet_bootstrap.cc:654] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:25.871333 11276 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: No bootstrap required, opened a new log
I20260929 11:33:25.871641 11276 ts_tablet_manager.cc:1397] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 11:33:25.872272 11278 log.cc:824] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:25.872853 11277 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: No bootstrap required, opened a new log
I20260929 11:33:25.873149 11277 ts_tablet_manager.cc:1397] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260929 11:33:25.876039 11278 tablet_bootstrap.cc:492] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: No bootstrap required, opened a new log
I20260929 11:33:25.876296 11278 ts_tablet_manager.cc:1397] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 11:33:25.880577 11277 raft_consensus.cc:348] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.880977 11277 raft_consensus.cc:374] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:25.881067 11277 raft_consensus.cc:729] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8107582abf4f07b77db2b18f7f0733, State: Initialized, Role: FOLLOWER
I20260929 11:33:25.881641 11277 consensus_queue.cc:260] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.881644 11276 raft_consensus.cc:348] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.882054 11276 raft_consensus.cc:374] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:25.882154 11276 raft_consensus.cc:729] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b0963143104767802317d710dadda7, State: Initialized, Role: FOLLOWER
I20260929 11:33:25.882782 10997 heartbeater.cc:503] Master 127.6.9.254:43107 was elected leader, sending a full tablet report...
I20260929 11:33:25.882838 11276 consensus_queue.cc:260] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.883395 11277 ts_tablet_manager.cc:1428] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260929 11:33:25.884546 11128 heartbeater.cc:503] Master 127.6.9.254:43107 was elected leader, sending a full tablet report...
I20260929 11:33:25.886206 11278 raft_consensus.cc:348] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.886647 11278 raft_consensus.cc:374] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:25.886751 11278 raft_consensus.cc:729] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310917afa00c4dd3bce75c551d59cd43, State: Initialized, Role: FOLLOWER
I20260929 11:33:25.887378 11278 consensus_queue.cc:260] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:25.887704 11276 ts_tablet_manager.cc:1428] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 11:33:25.888965 11258 heartbeater.cc:503] Master 127.6.9.254:43107 was elected leader, sending a full tablet report...
I20260929 11:33:25.889398 11278 ts_tablet_manager.cc:1428] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
W20260929 11:33:25.909438 10998 tablet.cc:2369] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:26.015043 11284 raft_consensus.cc:482] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:26.015367 11284 raft_consensus.cc:504] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:26.017077 11284 leader_election.cc:288] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8107582abf4f07b77db2b18f7f0733 (127.6.9.193:42997), 04b0963143104767802317d710dadda7 (127.6.9.194:36713)
I20260929 11:33:26.020040 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.017488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36713 (local address 127.6.9.195:50301)
0929 11:33:26.017818 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:26.017838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:26.017940 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:26.018325 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:26.018329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:26.018354 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:26.019162 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:26.019166 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:26.019186 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:26.019236 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:26.019630 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:26.019633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:26.019637 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 11:33:26.019750 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 11:33:26.019797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260929 11:33:26.020074 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.017692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:50301 (local address 127.6.9.194:36713)
0929 11:33:26.017859 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.017865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.017886 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:33:26.018123 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.018130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.018192 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.018289 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.019430 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.019436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.019441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.019461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.019501 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.019518 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.019646 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.019769 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.019831 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:33:26.021112 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.017606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54517 (local address 127.6.9.193:42997)
0929 11:33:26.017916 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.017922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.019132 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 11:33:26.019217 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.019222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.019288 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.019393 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.020367 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.020373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.020377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.020397 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.020435 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.020477 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.020564 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.020792 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.020850 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 11:33:26.021234 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.018142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42997 (local address 127.6.9.195:54517)
0929 11:33:26.018853 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:26.018874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:26.018988 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:26.019446 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:26.019451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:26.019476 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:26.020136 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:33:26.020139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:26.020153 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:33:26.020196 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:26.020650 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:26.020655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:26.020681 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:33:26.020784 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:33:26.020849 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":413,"threads_started":1}
I20260929 11:33:26.021085 11083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f242392e4cd4fe5a96df22e6039e718" candidate_uuid: "310917afa00c4dd3bce75c551d59cd43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b0963143104767802317d710dadda7" is_pre_election: true
I20260929 11:33:26.021709 11083 raft_consensus.cc:2454] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310917afa00c4dd3bce75c551d59cd43 in term 0.
I20260929 11:33:26.022167 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f242392e4cd4fe5a96df22e6039e718" candidate_uuid: "310917afa00c4dd3bce75c551d59cd43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8107582abf4f07b77db2b18f7f0733" is_pre_election: true
I20260929 11:33:26.022393 11146 leader_election.cc:302] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b0963143104767802317d710dadda7, 310917afa00c4dd3bce75c551d59cd43; no voters:
I20260929 11:33:26.022627 10952 raft_consensus.cc:2454] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310917afa00c4dd3bce75c551d59cd43 in term 0.
I20260929 11:33:26.022855 11284 raft_consensus.cc:2781] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:26.023005 11284 raft_consensus.cc:482] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:26.023116 11284 raft_consensus.cc:3037] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:26.025450 11284 raft_consensus.cc:504] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:26.026019 11284 leader_election.cc:288] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8107582abf4f07b77db2b18f7f0733 (127.6.9.193:42997), 04b0963143104767802317d710dadda7 (127.6.9.194:36713)
I20260929 11:33:26.026441 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f242392e4cd4fe5a96df22e6039e718" candidate_uuid: "310917afa00c4dd3bce75c551d59cd43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8107582abf4f07b77db2b18f7f0733"
I20260929 11:33:26.026556 11083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f242392e4cd4fe5a96df22e6039e718" candidate_uuid: "310917afa00c4dd3bce75c551d59cd43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b0963143104767802317d710dadda7"
I20260929 11:33:26.026651 10952 raft_consensus.cc:3037] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 0 FOLLOWER]: Advancing to term 1
W20260929 11:33:26.026724 11259 tablet.cc:2369] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:26.026783 11083 raft_consensus.cc:3037] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:26.029030 11083 raft_consensus.cc:2454] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310917afa00c4dd3bce75c551d59cd43 in term 1.
I20260929 11:33:26.029038 10952 raft_consensus.cc:2454] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310917afa00c4dd3bce75c551d59cd43 in term 1.
I20260929 11:33:26.029404 11146 leader_election.cc:302] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b0963143104767802317d710dadda7, 310917afa00c4dd3bce75c551d59cd43; no voters:
I20260929 11:33:26.029677 11284 raft_consensus.cc:2781] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:26.030102 11284 raft_consensus.cc:686] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 1 LEADER]: Becoming Leader. State: Replica: 310917afa00c4dd3bce75c551d59cd43, State: Running, Role: LEADER
I20260929 11:33:26.030551 11284 consensus_queue.cc:237] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } }
I20260929 11:33:26.034849 10811 catalog_manager.cc:5495] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 310917afa00c4dd3bce75c551d59cd43 (127.6.9.195). New cstate: current_term: 1 leader_uuid: "310917afa00c4dd3bce75c551d59cd43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310917afa00c4dd3bce75c551d59cd43" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:26.073262 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.071323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.6.9.195:39991)
0929 11:33:26.071512 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.071517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.071536 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:33:26.071673 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.071677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.071755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.071838 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.072708 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.072714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.072719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.072741 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.072773 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.072786 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.072893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.072963 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.073010 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:33:26.081302 10952 raft_consensus.cc:1261] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 1 FOLLOWER]: Refusing update from remote peer 310917afa00c4dd3bce75c551d59cd43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:26.081301 11083 raft_consensus.cc:1261] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 1 FOLLOWER]: Refusing update from remote peer 310917afa00c4dd3bce75c551d59cd43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:26.082254 11286 consensus_queue.cc:1035] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f8107582abf4f07b77db2b18f7f0733" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:26.082590 11284 consensus_queue.cc:1035] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b0963143104767802317d710dadda7" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:33:26.086252 11129 tablet.cc:2369] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:26.093571 11291 mvcc.cc:205] Tried to move back new op lower bound from 7334631858499309568 to 7334631858304565248. Current Snapshot: MvccSnapshot[applied={T|T < 7334631858499309568}]
I20260929 11:33:26.111835 10810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:26.112102 10810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:26.116082 10810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32a6d57c4cb34cd18531ee17f3fc2363: Sending DeleteTablet for 3 replicas of tablet 8f242392e4cd4fe5a96df22e6039e718
I20260929 11:33:26.117472 10932 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:26 UTC) from {username='slave'} at 127.0.0.1:34260
I20260929 11:33:26.117477 11193 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:26 UTC) from {username='slave'} at 127.0.0.1:59898
I20260929 11:33:26.118678 11063 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f242392e4cd4fe5a96df22e6039e718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:26 UTC) from {username='slave'} at 127.0.0.1:46138
I20260929 11:33:26.120952 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10870
I20260929 11:33:26.127251 11300 tablet_replica.cc:330] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: stopping tablet replica
I20260929 11:33:26.127794 11300 raft_consensus.cc:2229] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:26.128371 11300 raft_consensus.cc:2258] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:26.130824 11300 ts_tablet_manager.cc:1905] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:26.134672 11301 tablet_replica.cc:330] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: stopping tablet replica
I20260929 11:33:26.135309 11301 raft_consensus.cc:2229] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:26.136029 11301 raft_consensus.cc:2258] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:26.138785 11301 ts_tablet_manager.cc:1905] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:26.138818 11300 ts_tablet_manager.cc:1918] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:26.139016 11300 log.cc:1196] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/8f242392e4cd4fe5a96df22e6039e718
I20260929 11:33:26.139611 11300 ts_tablet_manager.cc:1939] T 8f242392e4cd4fe5a96df22e6039e718 P 5f8107582abf4f07b77db2b18f7f0733: Deleting consensus metadata
I20260929 11:33:26.141175 11299 tablet_replica.cc:330] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: stopping tablet replica
I20260929 11:33:26.141647 11299 raft_consensus.cc:2229] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:26.142342 11299 raft_consensus.cc:2258] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:26.143836 10798 catalog_manager.cc:4841] TS 5f8107582abf4f07b77db2b18f7f0733 (127.6.9.193:42997): tablet 8f242392e4cd4fe5a96df22e6039e718 (table test-table [id=40a211cd133145da9a3d756e2b51dee4]) successfully deleted
I20260929 11:33:26.144783 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.122116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.6.9.193:42997)
0929 11:33:26.122296 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.122302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.140833 (+ 18531us) server_negotiation.cc:407] Connection header received
0929 11:33:26.141316 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.141322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.141422 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.141494 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.142896 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.142901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.142906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.142927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.142965 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.142980 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.143084 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.144399 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.144465 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:33:26.145016 11299 ts_tablet_manager.cc:1905] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:26.145839 10992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:26.150533 11301 ts_tablet_manager.cc:1918] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:26.150719 11301 log.cc:1196] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/8f242392e4cd4fe5a96df22e6039e718
I20260929 11:33:26.151223 11301 ts_tablet_manager.cc:1939] T 8f242392e4cd4fe5a96df22e6039e718 P 04b0963143104767802317d710dadda7: Deleting consensus metadata
I20260929 11:33:26.152860 11299 ts_tablet_manager.cc:1918] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:26.153059 11299 log.cc:1196] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/8f242392e4cd4fe5a96df22e6039e718
I20260929 11:33:26.153226 10795 catalog_manager.cc:4841] TS 04b0963143104767802317d710dadda7 (127.6.9.194:36713): tablet 8f242392e4cd4fe5a96df22e6039e718 (table test-table [id=40a211cd133145da9a3d756e2b51dee4]) successfully deleted
I20260929 11:33:26.153609 11299 ts_tablet_manager.cc:1939] T 8f242392e4cd4fe5a96df22e6039e718 P 310917afa00c4dd3bce75c551d59cd43: Deleting consensus metadata
I20260929 11:33:26.155396 10796 catalog_manager.cc:4841] TS 310917afa00c4dd3bce75c551d59cd43 (127.6.9.195:39991): tablet 8f242392e4cd4fe5a96df22e6039e718 (table test-table [id=40a211cd133145da9a3d756e2b51dee4]) successfully deleted
I20260929 11:33:26.241791 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10870
I20260929 11:33:26.256078 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11001
I20260929 11:33:26.259016 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.256938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.6.9.194:36713)
0929 11:33:26.257151 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.257158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.257180 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:33:26.257248 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.257254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.257386 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.257498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.258372 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.258379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.258384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.258404 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.258442 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.258458 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.258584 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.258646 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.258707 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:33:26.259841 11123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:26.360755 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11001
I20260929 11:33:26.383306 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11131
I20260929 11:33:26.386703 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:26.384209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.6.9.195:39991)
0929 11:33:26.384411 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:26.384417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:26.384535 (+ 118us) server_negotiation.cc:407] Connection header received
0929 11:33:26.384778 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:26.384784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:26.384856 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:26.384975 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:26.385993 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:26.386003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:26.386008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:26.386028 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:26.386065 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:26.386081 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:26.386231 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:26.386303 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 11:33:26.386362 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:33:26.387544 11253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:26.506013 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11131
I20260929 11:33:26.521708 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 10780
I20260929 11:33:26.522799 10841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:26.626811 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 10780
2026-09-29T11:33:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2704 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 11:33:26.663467 6183 mini_oidc.cc:152] Starting JWKS server
I20260929 11:33:26.666203 6183 webserver.cc:460] Webserver started at https://127.0.0.1:33675/ using document root <none> and password file <none>
I20260929 11:33:26.666522 6183 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:33:26.667197 6183 webserver.cc:460] Webserver started at http://127.0.0.1:41853/ using document root <none> and password file <none>
2026-09-29T11:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:26Z Disabled control of system clock
I20260929 11:33:26.690912 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43273
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33675/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/testchainca.cert with env {}
W20260929 11:33:27.005971 11312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:27.006271 11312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:27.006359 11312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:27.013572 11312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:27.013700 11312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:27.013736 11312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:27.013787 11312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:27.013829 11312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:33:27.013897 11312 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:33:27.013935 11312 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33675/jwks.json
I20260929 11:33:27.023977 11312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33675/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:27.025303 11312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:27.026541 11312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:27.038123 11317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:27.038211 11318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:27.038228 11320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:27.038801 11312 server_base.cc:1034] running on GCE node
I20260929 11:33:27.039628 11312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:27.041148 11312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:27.042356 11312 hybrid_clock.cc:648] HybridClock initialized: now 1790681607042300 us; error 44 us; skew 500 ppm
I20260929 11:33:27.042708 11312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:27.044863 11312 webserver.cc:460] Webserver started at http://127.6.9.254:46619/ using document root <none> and password file <none>
I20260929 11:33:27.045532 11312 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:27.045636 11312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:27.045919 11312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:27.048750 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "865dada9d94b4379b1299509c5fdcea6"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "e9d48f968d3bfa4b200152b22a4ae671"
server_key_iv: "68464b29a5815a559250d319d671e176"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.049511 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "865dada9d94b4379b1299509c5fdcea6"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "e9d48f968d3bfa4b200152b22a4ae671"
server_key_iv: "68464b29a5815a559250d319d671e176"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.054279 11312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 11:33:27.057494 11326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:27.058880 11312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:27.059051 11312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "865dada9d94b4379b1299509c5fdcea6"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "e9d48f968d3bfa4b200152b22a4ae671"
server_key_iv: "68464b29a5815a559250d319d671e176"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.059258 11312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:27.086762 11312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:27.087575 11312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:27.087827 11312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:27.104413 11312 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43273
I20260929 11:33:27.104432 11377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43273 every 8 connection(s)
I20260929 11:33:27.105733 11312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:27.108897 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11312
I20260929 11:33:27.109104 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:27.109763 11378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:27.109869 6183 external_mini_cluster.cc:1428] Setting key c3fea5bca711d0610a2b78980060cc5b
I20260929 11:33:27.123615 11378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: Bootstrap starting.
I20260929 11:33:27.123790 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:27.111769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.6.9.254:43273)
0929 11:33:27.112735 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:27.112747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:27.112797 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:33:27.114470 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:27.114493 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:27.114969 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:27.115306 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:27.116028 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.117444 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.119346 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.120070 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.120242 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.121475 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:27.121485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:27.121501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:27.122056 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:27.122139 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:27.122161 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:27.122248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:27.122474 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 11:33:27.122809 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
I20260929 11:33:27.126991 11378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:27.127524 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:33:27.128127 11378 log.cc:824] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:27.130990 11378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: No bootstrap required, opened a new log
I20260929 11:33:27.138103 11378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } }
I20260929 11:33:27.138478 11378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:27.138581 11378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865dada9d94b4379b1299509c5fdcea6, State: Initialized, Role: FOLLOWER
I20260929 11:33:27.139159 11378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } }
I20260929 11:33:27.139389 11378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:27.139518 11378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:27.139663 11378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:27.141978 11378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } }
I20260929 11:33:27.142449 11378 leader_election.cc:302] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 865dada9d94b4379b1299509c5fdcea6; no voters:
I20260929 11:33:27.142961 11378 leader_election.cc:288] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:27.143105 11383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:27.143730 11383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [term 1 LEADER]: Becoming Leader. State: Replica: 865dada9d94b4379b1299509c5fdcea6, State: Running, Role: LEADER
I20260929 11:33:27.144227 11383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } }
I20260929 11:33:27.144969 11378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:27.146725 11384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "865dada9d94b4379b1299509c5fdcea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } } }
I20260929 11:33:27.147135 11384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:27.148028 11385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865dada9d94b4379b1299509c5fdcea6. Latest consensus state: current_term: 1 leader_uuid: "865dada9d94b4379b1299509c5fdcea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865dada9d94b4379b1299509c5fdcea6" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43273 } } }
I20260929 11:33:27.148310 11385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6 [sys.catalog]: This master's current role is: LEADER
W20260929 11:33:27.154088 11399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:33:27.154201 11399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:33:27.154350 11392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:27.155925 11392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:27.164898 11392 catalog_manager.cc:1347] Generated new cluster ID: ac7e3a74aad443adae1ca7a919000d58
I20260929 11:33:27.165038 11392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:27.191694 11392 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:27.193606 11392 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:27.207453 11392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865dada9d94b4379b1299509c5fdcea6: Generated new TSK 0
I20260929 11:33:27.208573 11392 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:33:27.252313 11408 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 11:33:27.252487 11408 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:33:27.253783 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:27.230708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.6.9.254:43273)
0929 11:33:27.230914 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:27.230920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:27.231461 (+ 541us) server_negotiation.cc:407] Connection header received
0929 11:33:27.231774 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:27.231780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:27.231899 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:27.232075 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:33:27.232712 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.233831 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.235270 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.236003 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.236137 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.236908 (+ 771us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:33:27.253030 (+ 16122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:33:27.253455 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 11:33:27.253937 11381 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 11:33:27.255209 11407 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:27.230676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43273 (local address 127.0.0.1:57120)
0929 11:33:27.231358 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:27.231379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:27.231532 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:27.232163 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:27.232168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:27.232182 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:33:27.232522 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:27.232534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.233986 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.233990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:27.235118 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:27.235139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.236236 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.236305 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:27.236460 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.236531 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:33:27.253396 (+ 16865us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:33:27.253685 (+ 289us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.9.254:43273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.9.254:43273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 11:33:27.257059 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11312
I20260929 11:33:27.258240 11373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:27.354657 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11312
2026-09-29T11:33:27Z chronyd exiting
I20260929 11:33:27.401927 6183 test_util.cc:175] -----------------------------------------------
I20260929 11:33:27.402081 6183 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681566518919-6183-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 11:33:27.405947 6183 mini_oidc.cc:152] Starting JWKS server
I20260929 11:33:27.407745 6183 webserver.cc:460] Webserver started at https://127.0.0.1:42905/ using document root <none> and password file <none>
I20260929 11:33:27.407958 6183 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:33:27.408548 6183 webserver.cc:460] Webserver started at http://127.0.0.1:39881/ using document root <none> and password file <none>
2026-09-29T11:33:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:27Z Disabled control of system clock
I20260929 11:33:27.432257 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40439
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40439
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/testchainca.cert with env {}
W20260929 11:33:27.743153 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:27.743425 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:27.743551 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:27.750517 11415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:27.750633 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:27.750705 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:27.750774 11415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:27.750818 11415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:33:27.750902 11415 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:33:27.750954 11415 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42905/jwks.json
I20260929 11:33:27.760869 11415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:27.762118 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:27.763417 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:27.774307 11420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:27.774503 11423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:27.774537 11421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:27.775499 11415 server_base.cc:1034] running on GCE node
I20260929 11:33:27.776230 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:27.777680 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:27.778909 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790681607778857 us; error 47 us; skew 500 ppm
I20260929 11:33:27.779255 11415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:27.781637 11415 webserver.cc:460] Webserver started at http://127.6.9.254:46851/ using document root <none> and password file <none>
I20260929 11:33:27.782361 11415 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:27.782475 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:27.782923 11415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:27.785895 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "022841fcca8148ed8f092c15b16872dd"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "5eca47c277edbe465d23173c1004ccd3"
server_key_iv: "df4fcf4d2db9261738a40028a035b6b0"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.786674 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "022841fcca8148ed8f092c15b16872dd"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "5eca47c277edbe465d23173c1004ccd3"
server_key_iv: "df4fcf4d2db9261738a40028a035b6b0"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.791699 11415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:27.795037 11429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:27.796363 11415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:27.796520 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "022841fcca8148ed8f092c15b16872dd"
format_stamp: "Formatted at 2026-09-29 11:33:27 on dist-test-slave-c0l9"
server_key: "5eca47c277edbe465d23173c1004ccd3"
server_key_iv: "df4fcf4d2db9261738a40028a035b6b0"
server_key_version: "encryptionkey@0"
I20260929 11:33:27.796744 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:27.822917 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:27.823765 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:27.824024 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:27.841001 11415 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40439
I20260929 11:33:27.841008 11480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40439 every 8 connection(s)
I20260929 11:33:27.842320 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:27.846102 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:27.850827 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11415
I20260929 11:33:27.851022 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:27.851819 6183 external_mini_cluster.cc:1428] Setting key 74e06de85dc7946c77093d163a2ee6f9
I20260929 11:33:27.857908 11481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd: Bootstrap starting.
I20260929 11:33:27.861588 11481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:27.862830 11481 log.cc:824] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:27.866050 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:27.853799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.6.9.254:40439)
0929 11:33:27.854790 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:27.854801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:27.854855 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:33:27.856581 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:27.856605 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:27.857161 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:27.857504 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:27.858129 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.859483 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.861284 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.862050 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.862195 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.863527 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:27.863538 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:27.863560 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:27.864340 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:27.864433 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:27.864456 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:27.864542 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:27.864795 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 11:33:27.865097 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":325,"threads_started":1}
I20260929 11:33:27.866179 11481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd: No bootstrap required, opened a new log
I20260929 11:33:27.869817 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:33:27.874168 11481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } }
I20260929 11:33:27.874508 11481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:27.874581 11481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022841fcca8148ed8f092c15b16872dd, State: Initialized, Role: FOLLOWER
I20260929 11:33:27.875108 11481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } }
I20260929 11:33:27.875321 11481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:27.875415 11481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:27.875541 11481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:27.877835 11481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } }
I20260929 11:33:27.878279 11481 leader_election.cc:302] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 022841fcca8148ed8f092c15b16872dd; no voters:
I20260929 11:33:27.878762 11481 leader_election.cc:288] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:27.878908 11486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:27.879884 11486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [term 1 LEADER]: Becoming Leader. State: Replica: 022841fcca8148ed8f092c15b16872dd, State: Running, Role: LEADER
I20260929 11:33:27.880435 11486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } }
I20260929 11:33:27.880903 11481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:27.883343 11488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022841fcca8148ed8f092c15b16872dd. Latest consensus state: current_term: 1 leader_uuid: "022841fcca8148ed8f092c15b16872dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } } }
I20260929 11:33:27.883373 11487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "022841fcca8148ed8f092c15b16872dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022841fcca8148ed8f092c15b16872dd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40439 } } }
I20260929 11:33:27.883726 11488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:27.883731 11487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:27.887405 11494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:27.892709 11494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:27.900910 11494 catalog_manager.cc:1347] Generated new cluster ID: d8df452f9d41489094754dceaa85f069
I20260929 11:33:27.901036 11494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:27.918342 11494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:27.920241 11494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:27.936894 11494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022841fcca8148ed8f092c15b16872dd: Generated new TSK 0
I20260929 11:33:27.938082 11494 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:33:27.979305 11511 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 11:33:27.979434 11511 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:33:27.980737 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:27.960396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.6.9.254:40439)
0929 11:33:27.960548 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:27.960552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:27.960955 (+ 403us) server_negotiation.cc:407] Connection header received
0929 11:33:27.961192 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:27.961199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:27.961304 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:27.961473 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:33:27.961946 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.963062 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.964536 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.965238 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.965363 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.965830 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:33:27.980015 (+ 14185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:33:27.980404 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 11:33:27.980829 11510 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:27.960248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40439 (local address 127.0.0.1:57550)
0929 11:33:27.960857 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:27.960873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:27.961003 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:27.961507 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:27.961510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:27.961521 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:33:27.961785 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:27.961793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.963225 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.963229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:27.964387 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:27.964400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:27.965350 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:27.965406 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:27.965524 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:27.965559 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:33:27.980268 (+ 14709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:33:27.980477 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.9.254:40439: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":308,"threads_started":1}
W20260929 11:33:27.980871 11484 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 11:33:27.982117 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11415
I20260929 11:33:27.983317 11476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:28.077015 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11415
2026-09-29T11:33:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 11:33:28.128960 6183 mini_oidc.cc:152] Starting JWKS server
I20260929 11:33:28.130801 6183 webserver.cc:460] Webserver started at https://127.0.0.1:45945/ using document root <none> and password file <none>
I20260929 11:33:28.131048 6183 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:33:28.131672 6183 webserver.cc:460] Webserver started at http://127.0.0.1:33705/ using document root <none> and password file <none>
2026-09-29T11:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:28Z Disabled control of system clock
I20260929 11:33:28.154678 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40127
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:33:28.462257 11518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:28.462586 11518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:28.462674 11518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:28.470069 11518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:28.470187 11518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:28.470223 11518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:28.470285 11518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:28.470329 11518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:33:28.470391 11518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:33:28.470431 11518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45945/jwks.json
I20260929 11:33:28.479709 11518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:28.481015 11518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:28.482772 11518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:28.494050 11523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:28.494087 11524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:28.493986 11526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:28.494871 11518 server_base.cc:1034] running on GCE node
I20260929 11:33:28.495664 11518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:28.497133 11518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:28.498346 11518 hybrid_clock.cc:648] HybridClock initialized: now 1790681608498290 us; error 52 us; skew 500 ppm
I20260929 11:33:28.498797 11518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:28.501210 11518 webserver.cc:460] Webserver started at http://127.6.9.254:33231/ using document root <none> and password file <none>
I20260929 11:33:28.501940 11518 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:28.502039 11518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:28.502367 11518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:28.505410 11518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "f7e19e7e277a4ba0854347b9e38469b5"
format_stamp: "Formatted at 2026-09-29 11:33:28 on dist-test-slave-c0l9"
server_key: "b9f36c97732371ffc4a58b2b04473952"
server_key_iv: "d93107c75c29100cbb13151130efd6d3"
server_key_version: "encryptionkey@0"
I20260929 11:33:28.506242 11518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "f7e19e7e277a4ba0854347b9e38469b5"
format_stamp: "Formatted at 2026-09-29 11:33:28 on dist-test-slave-c0l9"
server_key: "b9f36c97732371ffc4a58b2b04473952"
server_key_iv: "d93107c75c29100cbb13151130efd6d3"
server_key_version: "encryptionkey@0"
I20260929 11:33:28.511977 11518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:28.515576 11532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:28.517091 11518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:33:28.517282 11518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "f7e19e7e277a4ba0854347b9e38469b5"
format_stamp: "Formatted at 2026-09-29 11:33:28 on dist-test-slave-c0l9"
server_key: "b9f36c97732371ffc4a58b2b04473952"
server_key_iv: "d93107c75c29100cbb13151130efd6d3"
server_key_version: "encryptionkey@0"
I20260929 11:33:28.517498 11518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:28.538012 11518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:28.538898 11518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:28.539182 11518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:28.557217 11518 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40127
I20260929 11:33:28.557237 11583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40127 every 8 connection(s)
I20260929 11:33:28.558609 11518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:28.560091 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11518
I20260929 11:33:28.560302 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:28.561098 6183 external_mini_cluster.cc:1428] Setting key 93d946bd59095bd5ee8fa1012e6d1378
I20260929 11:33:28.563268 11584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:28.574421 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:28.562950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.6.9.254:40127)
0929 11:33:28.564008 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:28.564020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:28.564062 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:33:28.565728 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:28.565752 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:28.566262 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:28.566660 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:28.567240 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.568610 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.569912 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.570570 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.570737 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:28.572030 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:28.572042 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:28.572060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:28.572700 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:28.572787 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:28.572812 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:28.572917 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:28.573132 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 11:33:28.573457 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":408,"threads_started":1}
I20260929 11:33:28.574623 11584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5: Bootstrap starting.
I20260929 11:33:28.577914 11584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:28.578145 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:33:28.578929 11584 log.cc:824] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:28.581825 11584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5: No bootstrap required, opened a new log
I20260929 11:33:28.589056 11584 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } }
I20260929 11:33:28.589452 11584 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:28.589560 11584 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e19e7e277a4ba0854347b9e38469b5, State: Initialized, Role: FOLLOWER
I20260929 11:33:28.590152 11584 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } }
I20260929 11:33:28.590389 11584 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:28.590533 11584 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:28.590709 11584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:28.592918 11584 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } }
I20260929 11:33:28.593436 11584 leader_election.cc:302] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e19e7e277a4ba0854347b9e38469b5; no voters:
I20260929 11:33:28.593974 11584 leader_election.cc:288] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:28.594135 11589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:28.595033 11589 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [term 1 LEADER]: Becoming Leader. State: Replica: f7e19e7e277a4ba0854347b9e38469b5, State: Running, Role: LEADER
I20260929 11:33:28.595588 11589 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } }
I20260929 11:33:28.596169 11584 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:28.598335 11590 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e19e7e277a4ba0854347b9e38469b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } } }
I20260929 11:33:28.598889 11590 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:28.598655 11591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e19e7e277a4ba0854347b9e38469b5. Latest consensus state: current_term: 1 leader_uuid: "f7e19e7e277a4ba0854347b9e38469b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e19e7e277a4ba0854347b9e38469b5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40127 } } }
I20260929 11:33:28.599188 11591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:28.600880 11598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:28.606148 11598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:28.614864 11598 catalog_manager.cc:1347] Generated new cluster ID: 0ec420a3768d4984b636f89c8ef3cad5
I20260929 11:33:28.614989 11598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:28.630497 11598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:28.632367 11598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:28.645094 11598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7e19e7e277a4ba0854347b9e38469b5: Generated new TSK 0
I20260929 11:33:28.646171 11598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:33:28.678434 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:28.670268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.6.9.254:40127)
0929 11:33:28.670450 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:28.670457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:28.670939 (+ 482us) server_negotiation.cc:407] Connection header received
0929 11:33:28.671147 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:28.671153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:28.671243 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:28.671407 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:33:28.671804 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.672929 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.674255 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.674934 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.675056 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:28.675528 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:33:28.677800 (+ 2272us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:33:28.678125 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:33:28.678586 11587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 11:33:28.680145 11613 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:28.670234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40127 (local address 127.0.0.1:60514)
0929 11:33:28.670841 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:28.670857 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:28.670986 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:28.671404 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:28.671407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:28.671416 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:33:28.671670 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:28.671678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.673059 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.673063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:28.674099 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:28.674111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:28.675055 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:28.675115 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:28.675234 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:28.675266 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:33:28.678013 (+ 2747us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:33:28.679732 (+ 1719us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.9.254:40127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45945/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":1408,"thread_start_us":337,"threads_started":1}
I20260929 11:33:28.681469 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11518
I20260929 11:33:28.682688 11579 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 11:33:28.689937 11614 webserver.cc:533] Webserver: sslize failed:
E20260929 11:33:28.690028 11614 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:33:28.772687 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11518
2026-09-29T11:33:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (695 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T11:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:28Z Disabled control of system clock
I20260929 11:33:28.847879 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43563
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:37095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43563
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/testchainca.cert with env {}
W20260929 11:33:29.158315 11619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:29.158648 11619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:29.158778 11619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:29.166198 11619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:29.166317 11619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:29.166383 11619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:29.166457 11619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:29.166514 11619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:33:29.166574 11619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:33:29.166626 11619 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 11:33:29.176633 11619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:29.177982 11619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:29.179265 11619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:29.190670 11624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:29.190606 11625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:29.190644 11627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:29.191541 11619 server_base.cc:1034] running on GCE node
I20260929 11:33:29.192291 11619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:29.193724 11619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:29.194917 11619 hybrid_clock.cc:648] HybridClock initialized: now 1790681609194861 us; error 52 us; skew 500 ppm
I20260929 11:33:29.195246 11619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:33:29.197399 11619 webserver.cc:460] Webserver started at http://127.6.9.254:46859/ using document root <none> and password file <none>
I20260929 11:33:29.198063 11619 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:29.198161 11619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:29.198459 11619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:29.201331 11619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "7fc9f30bc54246dcbdd1232a50a6a73d"
format_stamp: "Formatted at 2026-09-29 11:33:29 on dist-test-slave-c0l9"
server_key: "96ebea6f8824a2b7800dec04be3e985c"
server_key_iv: "ed805b30304f07bafbc9dbc7816b0828"
server_key_version: "encryptionkey@0"
I20260929 11:33:29.202122 11619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "7fc9f30bc54246dcbdd1232a50a6a73d"
format_stamp: "Formatted at 2026-09-29 11:33:29 on dist-test-slave-c0l9"
server_key: "96ebea6f8824a2b7800dec04be3e985c"
server_key_iv: "ed805b30304f07bafbc9dbc7816b0828"
server_key_version: "encryptionkey@0"
I20260929 11:33:29.207422 11619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:29.211195 11633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:29.212641 11619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:33:29.212848 11619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7fc9f30bc54246dcbdd1232a50a6a73d"
format_stamp: "Formatted at 2026-09-29 11:33:29 on dist-test-slave-c0l9"
server_key: "96ebea6f8824a2b7800dec04be3e985c"
server_key_iv: "ed805b30304f07bafbc9dbc7816b0828"
server_key_version: "encryptionkey@0"
I20260929 11:33:29.213061 11619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:29.236701 11619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:29.237586 11619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:29.237859 11619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:29.254606 11619 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43563
I20260929 11:33:29.254626 11684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43563 every 8 connection(s)
I20260929 11:33:29.255898 11619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:29.259665 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:29.265416 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11619
I20260929 11:33:29.265596 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:29.266275 6183 external_mini_cluster.cc:1428] Setting key bcc1c045a20e889daa27c62e9414b276
I20260929 11:33:29.270490 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d: Bootstrap starting.
I20260929 11:33:29.273856 11685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:29.274852 11685 log.cc:824] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:29.277789 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d: No bootstrap required, opened a new log
I20260929 11:33:29.279781 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:29.268328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.6.9.254:43563)
0929 11:33:29.269392 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:29.269404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:29.269449 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:33:29.271132 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:29.271155 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:29.271622 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:29.271967 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:29.272614 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.273971 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.275310 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.276035 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.276218 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:29.277324 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:29.277332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:29.277347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:33:29.277891 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:29.277991 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:33:29.278018 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:29.278121 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:29.278356 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 11:33:29.278770 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":399,"threads_started":1}
I20260929 11:33:29.284104 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:33:29.287560 11685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } }
I20260929 11:33:29.287954 11685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:29.288061 11685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc9f30bc54246dcbdd1232a50a6a73d, State: Initialized, Role: FOLLOWER
I20260929 11:33:29.288733 11685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } }
I20260929 11:33:29.288960 11685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:29.289062 11685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:29.289191 11685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:29.291292 11685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } }
I20260929 11:33:29.291756 11685 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc9f30bc54246dcbdd1232a50a6a73d; no voters:
I20260929 11:33:29.292387 11685 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:29.292529 11690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:29.293174 11690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [term 1 LEADER]: Becoming Leader. State: Replica: 7fc9f30bc54246dcbdd1232a50a6a73d, State: Running, Role: LEADER
I20260929 11:33:29.293735 11690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } }
I20260929 11:33:29.294665 11685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:29.296851 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc9f30bc54246dcbdd1232a50a6a73d. Latest consensus state: current_term: 1 leader_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } } }
I20260929 11:33:29.296886 11691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9f30bc54246dcbdd1232a50a6a73d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43563 } } }
I20260929 11:33:29.297322 11691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:29.297322 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:29.299683 11695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:29.304893 11695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:29.313936 11695 catalog_manager.cc:1347] Generated new cluster ID: ac1f0f29c06647f5b1c39cf312f08c0d
I20260929 11:33:29.314065 11695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:29.347220 11695 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:29.349092 11695 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:29.363236 11695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc9f30bc54246dcbdd1232a50a6a73d: Generated new TSK 0
I20260929 11:33:29.364365 11695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:33:29.490031 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:29.383342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.6.9.254:43563)
0929 11:33:29.383536 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:29.383543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:29.383894 (+ 351us) server_negotiation.cc:407] Connection header received
0929 11:33:29.384109 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:29.384115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:29.384212 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:29.384386 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:33:29.384895 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.386016 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.387327 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.388012 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.388144 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:29.388578 (+ 434us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:33:29.489255 (+100677us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:33:29.489691 (+ 436us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38662: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 11:33:29.490180 11688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38662: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 11:33:29.490252 11714 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:29.383183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43563 (local address 127.0.0.1:38662)
0929 11:33:29.383803 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:29.383819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:29.383951 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:29.384385 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:29.384389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:29.384400 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:33:29.384720 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:29.384736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.386158 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.386162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:29.387190 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:29.387202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:29.388117 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:29.388160 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:29.388293 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:29.388325 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:33:29.489567 (+101242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:33:29.489868 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.9.254:43563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":357,"threads_started":1}
I20260929 11:33:29.492170 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11619
I20260929 11:33:29.493311 11680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:29.596256 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11619
2026-09-29T11:33:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (803 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (395 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (408 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:30 dist-test-slave-c0l9 krb5kdc[11729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:30 dist-test-slave-c0l9 krb5kdc[11729](info): set up 2 sockets
Sep 29 11:33:30 dist-test-slave-c0l9 krb5kdc[11729](info): commencing operation
krb5kdc: starting...
W20260929 11:33:31.396960 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:31 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:31.589812 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:37941
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:37941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:31.916453 11742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:31.916836 11742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:31.916985 11742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:31.924332 11742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:31.924475 11742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:31.924547 11742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:31.924624 11742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:31.924707 11742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:31.934660 11742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:37941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:31.936056 11742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:31.937422 11742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:31.949195 11747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:31.949062 11750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:31.949086 11748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:31.949808 11742 server_base.cc:1034] running on GCE node
I20260929 11:33:31.950639 11742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:31.952096 11742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:31.953307 11742 hybrid_clock.cc:648] HybridClock initialized: now 1790681611953243 us; error 51 us; skew 500 ppm
I20260929 11:33:31.953665 11742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:31 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681611, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:31.956771 11742 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:31.958338 11742 webserver.cc:460] Webserver started at http://127.6.9.254:42039/ using document root <none> and password file <none>
I20260929 11:33:31.959045 11742 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:31.959168 11742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:31.959491 11742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:31.962730 11742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "fe1e5767cd7746a1b4324ba139712aaf"
format_stamp: "Formatted at 2026-09-29 11:33:31 on dist-test-slave-c0l9"
server_key: "b1fbfa2a9e56e4b713f89d8809c33546"
server_key_iv: "d6a07ceee691a64e138f209e863ca6ea"
server_key_version: "encryptionkey@0"
I20260929 11:33:31.963591 11742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "fe1e5767cd7746a1b4324ba139712aaf"
format_stamp: "Formatted at 2026-09-29 11:33:31 on dist-test-slave-c0l9"
server_key: "b1fbfa2a9e56e4b713f89d8809c33546"
server_key_iv: "d6a07ceee691a64e138f209e863ca6ea"
server_key_version: "encryptionkey@0"
I20260929 11:33:31.969128 11742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:33:31.972949 11757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:31.974395 11742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:31.974581 11742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "fe1e5767cd7746a1b4324ba139712aaf"
format_stamp: "Formatted at 2026-09-29 11:33:31 on dist-test-slave-c0l9"
server_key: "b1fbfa2a9e56e4b713f89d8809c33546"
server_key_iv: "d6a07ceee691a64e138f209e863ca6ea"
server_key_version: "encryptionkey@0"
I20260929 11:33:31.974773 11742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:31.996459 11742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:32.000824 11742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:32.001103 11742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:32.018627 11742 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:37941
I20260929 11:33:32.018649 11808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:37941 every 8 connection(s)
I20260929 11:33:32.020061 11742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:32.023900 11809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:32.028841 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11742
I20260929 11:33:32.029039 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:32.029625 6183 external_mini_cluster.cc:1428] Setting key 9bd1d000b47cce9d39d2b7a223e91f6c
I20260929 11:33:32.034921 11809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf: Bootstrap starting.
I20260929 11:33:32.038312 11809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:32.039333 11809 log.cc:824] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:32.043001 11809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf: No bootstrap required, opened a new log
Sep 29 11:33:32 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:32.051127 11809 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } }
I20260929 11:33:32.051537 11809 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:32.051661 11809 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1e5767cd7746a1b4324ba139712aaf, State: Initialized, Role: FOLLOWER
I20260929 11:33:32.052302 11809 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } }
I20260929 11:33:32.052551 11809 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:32.052717 11809 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:32.052884 11809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:32.053188 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:32.031652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.6.9.254:37941)
0929 11:33:32.032595 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:32.032610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:32.032656 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:33:32.034448 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:32.034474 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:32.034892 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:32.035293 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:32.036787 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.038209 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.039538 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.040542 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.040786 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:32.045515 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:32.045526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:32.045547 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:32.045598 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:32.048792 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:32.049562 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:32.049570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:32.049577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:32.049766 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:32.050172 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:32.050176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:32.050178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:32.050765 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:32.051034 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:32.051562 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 11:33:32.052012 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":376,"threads_started":1}
I20260929 11:33:32.055651 11809 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } }
I20260929 11:33:32.056221 11809 leader_election.cc:302] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1e5767cd7746a1b4324ba139712aaf; no voters:
I20260929 11:33:32.056852 11809 leader_election.cc:288] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:32.056962 11814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:32.057590 11814 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [term 1 LEADER]: Becoming Leader. State: Replica: fe1e5767cd7746a1b4324ba139712aaf, State: Running, Role: LEADER
I20260929 11:33:32.058097 11814 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } }
I20260929 11:33:32.058750 11809 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:32.060691 11815 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1e5767cd7746a1b4324ba139712aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } } }
I20260929 11:33:32.061084 11815 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:32.062305 11816 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1e5767cd7746a1b4324ba139712aaf. Latest consensus state: current_term: 1 leader_uuid: "fe1e5767cd7746a1b4324ba139712aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1e5767cd7746a1b4324ba139712aaf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 37941 } } }
I20260929 11:33:32.062568 11816 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:32.065936 11822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:32.070564 11822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:32.079012 11822 catalog_manager.cc:1347] Generated new cluster ID: e63deb35a8504d6884e0e2c98531ba00
I20260929 11:33:32.079142 11822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:32.104031 11822 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:32.106005 11822 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:32.118386 11822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1e5767cd7746a1b4324ba139712aaf: Generated new TSK 0
I20260929 11:33:32.119554 11822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:32.164124 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:32.493503 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:32.493795 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:32.493914 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:32.501145 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:32.501291 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:32.501461 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:32.511307 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:32.512642 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:32.513988 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:32.526623 11840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:32.526513 11841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:32.526544 11843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:32.527611 11835 server_base.cc:1034] running on GCE node
I20260929 11:33:32.528244 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:32.529238 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:32.530431 11835 hybrid_clock.cc:648] HybridClock initialized: now 1790681612530385 us; error 38 us; skew 500 ppm
I20260929 11:33:32.530735 11835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:32 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681612, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:32.533776 11835 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:32.535131 11835 webserver.cc:460] Webserver started at http://127.6.9.193:35543/ using document root <none> and password file <none>
I20260929 11:33:32.535820 11835 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:32.535930 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:32.536186 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:32.539242 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "54f12cd0b03e4c3db5b1f4d266e10b55"
format_stamp: "Formatted at 2026-09-29 11:33:32 on dist-test-slave-c0l9"
server_key: "101e37e468082821cfea864d829ef91a"
server_key_iv: "c73b68d4c93921dad171fb25a4e08bcb"
server_key_version: "encryptionkey@0"
I20260929 11:33:32.540011 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "54f12cd0b03e4c3db5b1f4d266e10b55"
format_stamp: "Formatted at 2026-09-29 11:33:32 on dist-test-slave-c0l9"
server_key: "101e37e468082821cfea864d829ef91a"
server_key_iv: "c73b68d4c93921dad171fb25a4e08bcb"
server_key_version: "encryptionkey@0"
I20260929 11:33:32.545148 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 11:33:32.548398 11850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:32.549790 11835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:32.549926 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "54f12cd0b03e4c3db5b1f4d266e10b55"
format_stamp: "Formatted at 2026-09-29 11:33:32 on dist-test-slave-c0l9"
server_key: "101e37e468082821cfea864d829ef91a"
server_key_iv: "c73b68d4c93921dad171fb25a4e08bcb"
server_key_version: "encryptionkey@0"
I20260929 11:33:32.550093 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:32.566078 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:32.570766 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:32.571025 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:32.571949 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:32.573378 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:32.573451 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:32.573542 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:32.573585 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:32.602411 11835 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:38669
I20260929 11:33:32.602454 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:38669 every 8 connection(s)
I20260929 11:33:32.603667 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:32.613631 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11835
I20260929 11:33:32.613873 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:32.614607 6183 external_mini_cluster.cc:1428] Setting key 3a341dce4222020be5c0ac67a8b4d330
Sep 29 11:33:32 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681612, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:32.623981 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:32.606098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:53061 (local address 127.6.9.254:37941)
0929 11:33:32.606540 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:32.606547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:32.607491 (+ 944us) server_negotiation.cc:407] Connection header received
0929 11:33:32.609246 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:32.609250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:32.609320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:32.609417 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:32.610799 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.611515 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.613217 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.613851 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.614014 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:32.617431 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:32.617438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:32.617442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:32.617491 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:32.620629 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:32.621467 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:32.621473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:32.621476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:32.621645 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:32.622040 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:32.622045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:32.622049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:32.622444 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:32.622601 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:32.623277 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 11:33:32.623549 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":200,"threads_started":1}
I20260929 11:33:32.624711 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:32.606623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:37941 (local address 127.6.9.193:53061)
0929 11:33:32.607328 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:32.607372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:32.609010 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:32.609602 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:32.609609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:32.610119 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:32.610650 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:32.610666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.611704 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.611708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:32.613038 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:32.613051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:32.614098 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:32.614104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:32.614314 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:32.614989 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:32.615011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:32.617164 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:32.620785 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:32.620793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:32.620807 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:32.621333 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:32.621744 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:32.621747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:32.621749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:32.621936 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:32.622558 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:32.622563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:32.622894 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 11:33:32.623119 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 11:33:32.623525 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260929 11:33:32.626225 11963 heartbeater.cc:347] Connected to a master server at 127.6.9.254:37941
I20260929 11:33:32.626534 11963 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:32.627305 11963 heartbeater.cc:511] Master 127.6.9.254:37941 requested a full tablet report, sending...
I20260929 11:33:32.629468 11774 ts_manager.cc:194] Registered new tserver with Master: 54f12cd0b03e4c3db5b1f4d266e10b55 (127.6.9.193:38669)
I20260929 11:33:32.631042 11774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:53061
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:32.659390 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:33.001096 11969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:33.001381 11969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:33.001506 11969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:33.009177 11969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:33.009296 11969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:33.009444 11969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:33.019505 11969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:33.020818 11969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:33.022068 11969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:33.034844 11974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:33.034839 11977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:33.034780 11975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:33.035324 11969 server_base.cc:1034] running on GCE node
I20260929 11:33:33.035964 11969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:33.036906 11969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:33.038127 11969 hybrid_clock.cc:648] HybridClock initialized: now 1790681613038077 us; error 47 us; skew 500 ppm
I20260929 11:33:33.038422 11969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.041494 11969 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:33.042860 11969 webserver.cc:460] Webserver started at http://127.6.9.194:45377/ using document root <none> and password file <none>
I20260929 11:33:33.043501 11969 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:33.043601 11969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:33.043850 11969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:33.046797 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "fd45fb8eb27f489bb2ed4b404dfa4f24"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "ab19a64677db3c675257d6b672fc2b42"
server_key_iv: "5e65400207ead60026ce7cd3660fa37b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.047533 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "fd45fb8eb27f489bb2ed4b404dfa4f24"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "ab19a64677db3c675257d6b672fc2b42"
server_key_iv: "5e65400207ead60026ce7cd3660fa37b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.052502 11969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:33:33.055851 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.057140 11969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:33.057268 11969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "fd45fb8eb27f489bb2ed4b404dfa4f24"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "ab19a64677db3c675257d6b672fc2b42"
server_key_iv: "5e65400207ead60026ce7cd3660fa37b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.057430 11969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:33.074981 11969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:33.078951 11969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:33.079183 11969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:33.080108 11969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:33.081458 11969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:33.081532 11969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.081614 11969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:33.081655 11969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.110145 11969 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:33671
I20260929 11:33:33.110193 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:33671 every 8 connection(s)
I20260929 11:33:33.111349 11969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:33.119494 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 11969
I20260929 11:33:33.119654 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:33.120065 6183 external_mini_cluster.cc:1428] Setting key 81338c6c5df1164d787dfc9c58d60168
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:33.130312 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.113799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56897 (local address 127.6.9.254:37941)
0929 11:33:33.114010 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.114016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.115134 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 11:33:33.116827 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.116832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.116902 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.116997 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.118402 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.119165 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.120821 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.121369 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.121491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.124809 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.124814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.124817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.124850 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.127337 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.128037 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.128041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.128043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.128173 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.128588 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.128592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.128594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.128902 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.129042 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.129755 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.129994 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:33:33.131243 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.114297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:37941 (local address 127.6.9.194:56897)
0929 11:33:33.114975 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:33.115017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:33.116556 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:33.117180 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:33.117188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:33.117707 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:33.118265 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:33.118282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.119310 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.119314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:33.120652 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:33.120693 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.121562 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.121567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:33.121811 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.122475 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:33.122502 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:33.124573 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:33.127479 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:33.127488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:33.127505 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:33.127912 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:33.128275 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:33.128280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:33.128284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:33.128478 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:33.129028 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:33.129036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:33.129382 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 11:33:33.129630 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 11:33:33.130144 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
I20260929 11:33:33.132736 12097 heartbeater.cc:347] Connected to a master server at 127.6.9.254:37941
I20260929 11:33:33.133038 12097 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:33.133837 12097 heartbeater.cc:511] Master 127.6.9.254:37941 requested a full tablet report, sending...
I20260929 11:33:33.135387 11774 ts_manager.cc:194] Registered new tserver with Master: fd45fb8eb27f489bb2ed4b404dfa4f24 (127.6.9.194:33671)
I20260929 11:33:33.136566 11774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:56897
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:33.161969 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:33.478961 12102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:33.479259 12102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:33.479408 12102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:33.486708 12102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:33.486838 12102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:33.486995 12102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:33.496549 12102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:33.497864 12102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:33.499200 12102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:33.511981 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:33.512094 12107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:33.512023 12108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:33.513367 12102 server_base.cc:1034] running on GCE node
I20260929 11:33:33.514029 12102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:33.515013 12102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:33.516214 12102 hybrid_clock.cc:648] HybridClock initialized: now 1790681613516153 us; error 59 us; skew 500 ppm
I20260929 11:33:33.516538 12102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.519678 12102 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:33.521255 12102 webserver.cc:460] Webserver started at http://127.6.9.195:35899/ using document root <none> and password file <none>
I20260929 11:33:33.521941 12102 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:33.522068 12102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:33.522382 12102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:33.525419 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "35bf1315129e4898b70782b0a17d35d0"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "cef371ab97e833bd096004cc03b6500d"
server_key_iv: "105e3b5fc4012d7fd569b6ae8d29349b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.526206 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "35bf1315129e4898b70782b0a17d35d0"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "cef371ab97e833bd096004cc03b6500d"
server_key_iv: "105e3b5fc4012d7fd569b6ae8d29349b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.531386 12102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:33.534907 12117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.536198 12102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:33:33.536355 12102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "35bf1315129e4898b70782b0a17d35d0"
format_stamp: "Formatted at 2026-09-29 11:33:33 on dist-test-slave-c0l9"
server_key: "cef371ab97e833bd096004cc03b6500d"
server_key_iv: "105e3b5fc4012d7fd569b6ae8d29349b"
server_key_version: "encryptionkey@0"
I20260929 11:33:33.536558 12102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:33.563340 12102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:33.567404 12102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:33.567675 12102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:33.568612 12102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:33.569895 12102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:33.569981 12102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.570099 12102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:33.570152 12102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:33.599145 12102 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:35483
I20260929 11:33:33.599191 12229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:35483 every 8 connection(s)
I20260929 11:33:33.600340 12102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:33.604455 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12102
I20260929 11:33:33.604626 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:33.605193 6183 external_mini_cluster.cc:1428] Setting key e4d95b81bdc21997234a2ee6299c7a27
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:33.621167 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.602822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:53299 (local address 127.6.9.254:37941)
0929 11:33:33.603040 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.603047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.605298 (+ 2251us) server_negotiation.cc:407] Connection header received
0929 11:33:33.607591 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.607596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.607677 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.607787 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.609304 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.610070 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.611554 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.611984 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.612088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.615428 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.615433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.615436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.615470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.617918 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.618719 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.618723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.618726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.618890 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.619367 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.619371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.619373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.619718 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.619854 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.620724 (+ 870us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.620924 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:33:33.622184 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.603168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:37941 (local address 127.6.9.195:53299)
0929 11:33:33.605141 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:33.605190 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:33.607337 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:33.608014 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:33.608026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:33.608531 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:33.609150 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:33.609169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.610185 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.610189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:33.611411 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:33.611424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.612193 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.612198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:33.612404 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.613094 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:33.613116 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:33.615218 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:33.618026 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:33.618033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:33.618046 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:33.618582 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:33.619009 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:33.619012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:33.619014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:33.619254 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:33.619849 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:33.619857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:33.620273 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 11:33:33.620577 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 11:33:33.621056 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"thread_start_us":1524,"threads_started":1}
I20260929 11:33:33.623582 12230 heartbeater.cc:347] Connected to a master server at 127.6.9.254:37941
I20260929 11:33:33.623883 12230 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:33.624651 12230 heartbeater.cc:511] Master 127.6.9.254:37941 requested a full tablet report, sending...
I20260929 11:33:33.626147 11774 ts_manager.cc:194] Registered new tserver with Master: 35bf1315129e4898b70782b0a17d35d0 (127.6.9.195:35483)
I20260929 11:33:33.626950 11774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:53299
I20260929 11:33:33.633071 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:33:33.634236 11963 heartbeater.cc:503] Master 127.6.9.254:37941 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/test-admin.keytab.
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.657814 6183 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 11:33:33.663292 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:33.660752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.6.9.193:38669)
0929 11:33:33.661406 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.661415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.661444 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:33:33.661505 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.661530 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.661763 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.661916 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.662839 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58298: BlockingRecv error: recv got EOF from 127.0.0.1:58298 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":343,"threads_started":1}
W20260929 11:33:33.663415 6183 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.665096 6183 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.666416 6183 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:33.681751 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.668089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.6.9.193:38669)
0929 11:33:33.668281 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.668287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.668396 (+ 109us) server_negotiation.cc:407] Connection header received
0929 11:33:33.668572 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.668576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.668639 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.668742 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.669721 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.670695 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.671999 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.672635 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.672803 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.676352 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.676365 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.676381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.676418 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.679313 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.679935 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.679943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.679945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.680059 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.680350 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.680353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.680355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.680749 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.680957 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.681227 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.681466 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 11:33:33.687109 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:33.685611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.6.9.193:38669)
0929 11:33:33.685811 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.685816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.686000 (+ 184us) server_negotiation.cc:407] Connection header received
0929 11:33:33.686121 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.686125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.686191 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.686263 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.686870 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58312: BlockingRecv error: recv got EOF from 127.0.0.1:58312 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 11:33:33.687201 6183 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.688880 6183 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.690058 6183 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:33.703053 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.691232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.6.9.193:38669)
0929 11:33:33.691423 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.691428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.691632 (+ 204us) server_negotiation.cc:407] Connection header received
0929 11:33:33.691763 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.691766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.691822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.691893 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.692729 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.693429 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.694594 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.695015 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.695103 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.698523 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.698527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.698531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.698562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.700954 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.701538 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.701541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.701543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.701654 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.701945 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.701948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.701950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.702202 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.702381 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.702597 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.702751 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:33:33.707297 12237 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:33.705644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.6.9.193:38669)
0929 11:33:33.705775 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.705780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.706131 (+ 351us) server_negotiation.cc:407] Connection header received
0929 11:33:33.706287 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.706291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.706355 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.706429 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.707110 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58330: BlockingRecv error: recv got EOF from 127.0.0.1:58330 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 11:33:33.707477 6183 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681566518919-6183-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.708927 6183 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:33.710136 6183 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:33.723237 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.711279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.6.9.193:38669)
0929 11:33:33.711472 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.711478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.711678 (+ 200us) server_negotiation.cc:407] Connection header received
0929 11:33:33.711799 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.711802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.711857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.711932 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.712931 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.713629 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.714822 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.715215 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.715301 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.718655 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.718660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.718663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.718695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.721026 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.721635 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.721638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.721640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.721786 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.722049 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.722052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.722054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.722323 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.722544 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.722714 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.722876 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:33:33.724888 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11835
I20260929 11:33:33.735720 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.725578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.6.9.193:38669)
0929 11:33:33.725749 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.725756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.726065 (+ 309us) server_negotiation.cc:407] Connection header received
0929 11:33:33.726197 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.726200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.726260 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.726332 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.727160 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.727830 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.728987 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.729417 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.729515 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.731283 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.731288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.731291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.731319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.733678 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.734189 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.734192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.734194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.734315 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.734582 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.734585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.734588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.734845 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.735028 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.735255 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.735414 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:33:33.736269 11958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:33.818776 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11835
I20260929 11:33:33.831568 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11969
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:33.846570 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.832403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.6.9.194:33671)
0929 11:33:33.832964 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.832971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.832997 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:33:33.833064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.833071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.833272 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.833409 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.834425 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.835430 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.836631 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.837240 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.837342 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.841127 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.841141 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.841157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.841197 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.843875 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.844560 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.844567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.844569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.844812 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.845179 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.845182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.845184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.845537 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.845752 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.846012 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.846233 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
I20260929 11:33:33.847824 12092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:33.929854 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11969
I20260929 11:33:33.942983 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12102
Sep 29 11:33:33 dist-test-slave-c0l9 krb5kdc[11729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:33.958639 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:33.943966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.6.9.195:35483)
0929 11:33:33.944766 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:33.944773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:33.944810 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:33:33.944874 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:33.944880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:33.945097 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:33.945227 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:33.946227 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.947251 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.948514 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:33.949149 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:33.949273 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:33.953183 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:33.953198 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:33.953214 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:33.953264 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:33.955927 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.956567 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.956574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.956577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.956756 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:33.957201 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:33.957204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:33.957207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:33.957574 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:33.957823 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:33.958079 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 11:33:33.958318 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":425,"threads_started":1}
I20260929 11:33:33.959993 12225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:34.041324 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12102
I20260929 11:33:34.053624 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 11742
I20260929 11:33:34.054855 11804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:34.156826 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 11742
2026-09-29T11:33:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3761 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:34 dist-test-slave-c0l9 krb5kdc[12257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:34 dist-test-slave-c0l9 krb5kdc[12257](info): set up 2 sockets
Sep 29 11:33:34 dist-test-slave-c0l9 krb5kdc[12257](info): commencing operation
krb5kdc: starting...
W20260929 11:33:35.157051 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:35 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab.
Entry for principal oryx/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab.
I20260929 11:33:35.358130 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46741
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab
--principal=oryx/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:35.676792 12270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:35.677103 12270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:35.677232 12270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:35.684554 12270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:35.684690 12270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:35.684764 12270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:35.684844 12270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:35.684897 12270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:35.694796 12270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab
--principal=oryx/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:35.696161 12270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:35.697511 12270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:35.709200 12276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:35.709254 12275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:35.709232 12278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:35.710222 12270 server_base.cc:1034] running on GCE node
I20260929 11:33:35.711010 12270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:35.712457 12270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:35.713661 12270 hybrid_clock.cc:648] HybridClock initialized: now 1790681615713604 us; error 48 us; skew 500 ppm
I20260929 11:33:35.714020 12270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:35 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:35.720083 12270 init.cc:377] Logged in from keytab as oryx/127.6.9.254@KRBTEST.COM (short username oryx)
I20260929 11:33:35.721614 12270 webserver.cc:460] Webserver started at http://127.6.9.254:43993/ using document root <none> and password file <none>
I20260929 11:33:35.722316 12270 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:35.722436 12270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:35.722729 12270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:35.725788 12270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "715519d5829c48c48a4bcdfe3a49bfbf"
format_stamp: "Formatted at 2026-09-29 11:33:35 on dist-test-slave-c0l9"
server_key: "03cf75e7be034563ef8b540c375a2ade"
server_key_iv: "a6856943c012e8404d36cc7b8aa0b478"
server_key_version: "encryptionkey@0"
I20260929 11:33:35.726543 12270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "715519d5829c48c48a4bcdfe3a49bfbf"
format_stamp: "Formatted at 2026-09-29 11:33:35 on dist-test-slave-c0l9"
server_key: "03cf75e7be034563ef8b540c375a2ade"
server_key_iv: "a6856943c012e8404d36cc7b8aa0b478"
server_key_version: "encryptionkey@0"
I20260929 11:33:35.731614 12270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:35.734972 12285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:35.736316 12270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:33:35.736483 12270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "715519d5829c48c48a4bcdfe3a49bfbf"
format_stamp: "Formatted at 2026-09-29 11:33:35 on dist-test-slave-c0l9"
server_key: "03cf75e7be034563ef8b540c375a2ade"
server_key_iv: "a6856943c012e8404d36cc7b8aa0b478"
server_key_version: "encryptionkey@0"
I20260929 11:33:35.736680 12270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:35.759147 12270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:35.763216 12270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:35.763484 12270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:35.780496 12270 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46741
I20260929 11:33:35.780519 12336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46741 every 8 connection(s)
I20260929 11:33:35.781874 12270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:35.786012 12337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:35.788096 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12270
I20260929 11:33:35.788486 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:35.789234 6183 external_mini_cluster.cc:1428] Setting key 29e55fcd94296f49c5a17e261d7000f4
I20260929 11:33:35.798905 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Bootstrap starting.
I20260929 11:33:35.802295 12337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:35.803251 12337 log.cc:824] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:33:35 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:35.805976 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: No bootstrap required, opened a new log
I20260929 11:33:35.811774 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:35.791351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.6.9.254:46741)
0929 11:33:35.792173 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:35.792189 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:35.792238 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:33:35.794476 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:35.794504 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:35.795043 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:35.795506 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:35.796792 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:35.798691 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:35.799987 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:35.801067 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:35.801233 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:35.804975 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:35.804982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:35.804997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:35.805042 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:35.807909 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:35.808559 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:35.808565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:35.808574 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:35.808769 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:35.809127 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:35.809132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:35.809136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:35.809650 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:35.809929 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:35.810377 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 11:33:35.810788 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":333,"threads_started":1}
I20260929 11:33:35.813529 12337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } }
I20260929 11:33:35.813876 12337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:35.813977 12337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715519d5829c48c48a4bcdfe3a49bfbf, State: Initialized, Role: FOLLOWER
I20260929 11:33:35.814540 12337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } }
I20260929 11:33:35.814752 12337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:35.814870 12337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:35.815029 12337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:35.817131 12337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } }
I20260929 11:33:35.817608 12337 leader_election.cc:302] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 715519d5829c48c48a4bcdfe3a49bfbf; no voters:
I20260929 11:33:35.818154 12337 leader_election.cc:288] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:35.818272 12342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:35.818895 12342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 LEADER]: Becoming Leader. State: Replica: 715519d5829c48c48a4bcdfe3a49bfbf, State: Running, Role: LEADER
I20260929 11:33:35.819435 12342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } }
I20260929 11:33:35.820147 12337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:35.822374 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 715519d5829c48c48a4bcdfe3a49bfbf. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } }
I20260929 11:33:35.822753 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:35.822576 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } }
I20260929 11:33:35.822924 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:35.824278 12350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:35.828456 12350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:35.836781 12350 catalog_manager.cc:1347] Generated new cluster ID: e87b41d2600742d9a2565ab6f49a0807
I20260929 11:33:35.836905 12350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:35.851384 12350 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:35.852945 12350 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:35.861698 12350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Generated new TSK 0
I20260929 11:33:35.862604 12350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
I20260929 11:33:35.905764 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:36.226238 12363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:36.226512 12363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:36.226617 12363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:36.233675 12363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:36.233790 12363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:36.233911 12363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:36.243609 12363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:36.244871 12363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:36.246138 12363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:36.258724 12369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:36.258945 12371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:36.259124 12368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:36.259395 12363 server_base.cc:1034] running on GCE node
I20260929 11:33:36.259989 12363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:36.260957 12363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:36.262156 12363 hybrid_clock.cc:648] HybridClock initialized: now 1790681616262108 us; error 41 us; skew 500 ppm
I20260929 11:33:36.262450 12363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:36 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681616, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:36.265580 12363 init.cc:377] Logged in from keytab as oryx/127.6.9.193@KRBTEST.COM (short username oryx)
I20260929 11:33:36.267007 12363 webserver.cc:460] Webserver started at http://127.6.9.193:39633/ using document root <none> and password file <none>
I20260929 11:33:36.267692 12363 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:36.267809 12363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:36.268086 12363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:36.271003 12363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "38d8dae75ba16744fd9592064a4925f8"
server_key_iv: "43eaf68535ca38f0775a9d0af5130cda"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.271749 12363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "38d8dae75ba16744fd9592064a4925f8"
server_key_iv: "43eaf68535ca38f0775a9d0af5130cda"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.276791 12363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:33:36.280098 12378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.281384 12363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:36.281538 12363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "38d8dae75ba16744fd9592064a4925f8"
server_key_iv: "43eaf68535ca38f0775a9d0af5130cda"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.281739 12363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:36.310470 12363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:36.314819 12363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:36.315062 12363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:36.315949 12363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:36.317343 12363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:36.317416 12363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.317504 12363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:36.317555 12363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.346700 12363 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:42045
I20260929 11:33:36.346743 12490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:42045 every 8 connection(s)
I20260929 11:33:36.348032 12363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:36.354859 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12363
I20260929 11:33:36.355037 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:36.355822 6183 external_mini_cluster.cc:1428] Setting key 12f2f0cd718b4d6ed7bfb82c60630fd2
Sep 29 11:33:36 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681616, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:36.368309 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:36.350414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:35819 (local address 127.6.9.254:46741)
0929 11:33:36.350849 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:36.350855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:36.351805 (+ 950us) server_negotiation.cc:407] Connection header received
0929 11:33:36.353613 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:36.353618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:36.353685 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:36.353769 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:36.355218 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.356303 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.358299 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.358851 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.358988 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:36.362410 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:36.362416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:36.362421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:36.362460 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:36.365286 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:36.366106 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:36.366110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:36.366113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:36.366221 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:36.366645 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:36.366648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:36.366650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:36.366910 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:36.367023 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:36.367853 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 11:33:36.368051 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":201,"threads_started":1}
I20260929 11:33:36.369764 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:36.350990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:35819)
0929 11:33:36.351654 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:36.351691 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:36.353355 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:36.353953 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:36.353961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:36.354451 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:36.355006 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:36.355039 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.356446 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.356451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:36.358154 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:36.358167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.359028 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.359033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:36.359219 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:36.359834 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:36.359857 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:36.362183 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:36.365426 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:36.365435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:36.365451 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:36.366001 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:36.366308 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:36.366313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:36.366316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:36.366556 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:36.367028 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:36.367036 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:36.367490 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 11:33:36.367789 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 11:33:36.368325 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":261,"threads_started":1}
I20260929 11:33:36.371474 12491 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:36.371773 12491 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:36.372572 12491 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
I20260929 11:33:36.374742 12302 ts_manager.cc:194] Registered new tserver with Master: 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045)
WARNING: no policy specified for oryx/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:33:36.376190 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:35819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
I20260929 11:33:36.400597 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:36.724972 12497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:36.725287 12497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:36.725426 12497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:36.732877 12497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:36.733026 12497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:36.733168 12497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:36.743343 12497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:36.744765 12497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:36.746088 12497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:36.759040 12502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:36.759100 12505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:36.759011 12503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:36.760808 12497 server_base.cc:1034] running on GCE node
I20260929 11:33:36.761459 12497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:36.762446 12497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:36.763650 12497 hybrid_clock.cc:648] HybridClock initialized: now 1790681616763601 us; error 41 us; skew 500 ppm
I20260929 11:33:36.763957 12497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:36 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681616, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:36.767174 12497 init.cc:377] Logged in from keytab as oryx/127.6.9.194@KRBTEST.COM (short username oryx)
I20260929 11:33:36.768762 12497 webserver.cc:460] Webserver started at http://127.6.9.194:40031/ using document root <none> and password file <none>
I20260929 11:33:36.769501 12497 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:36.769632 12497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:36.769923 12497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:36.772972 12497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "b98d47edc02a42d59ef41d5553f37fb1"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "7ecbc84dcd249e818eb5d80f816d0f08"
server_key_iv: "e1bc6825d6ff9610630a3bf909f0068b"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.773774 12497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "b98d47edc02a42d59ef41d5553f37fb1"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "7ecbc84dcd249e818eb5d80f816d0f08"
server_key_iv: "e1bc6825d6ff9610630a3bf909f0068b"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.779268 12497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:36.783118 12512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.784487 12497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:33:36.784682 12497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "b98d47edc02a42d59ef41d5553f37fb1"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "7ecbc84dcd249e818eb5d80f816d0f08"
server_key_iv: "e1bc6825d6ff9610630a3bf909f0068b"
server_key_version: "encryptionkey@0"
I20260929 11:33:36.784921 12497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:36.796054 12497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:36.800110 12497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:36.800364 12497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:36.801301 12497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:36.802659 12497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:36.802749 12497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.802897 12497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:36.802950 12497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:36.833663 12497 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:33983
I20260929 11:33:36.833643 12624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:33983 every 8 connection(s)
I20260929 11:33:36.835498 12497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:36.840070 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12497
I20260929 11:33:36.840226 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:36.840770 6183 external_mini_cluster.cc:1428] Setting key 54e1e267e70eb4aba49ff225ab472522
Sep 29 11:33:36 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681616, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:36.857249 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:36.838262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:42961 (local address 127.6.9.254:46741)
0929 11:33:36.838480 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:36.838485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:36.839696 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 11:33:36.841791 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:36.841798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:36.841885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:36.841975 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:36.843672 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.844405 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.846049 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.846476 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.846597 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:36.850352 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:36.850357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:36.850361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:36.850397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:36.852877 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:36.853617 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:36.853622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:36.853624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:36.853892 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:36.854637 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:36.854643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:36.854646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:36.855115 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:36.855316 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:36.856506 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0929 11:33:36.856811 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:33:36.858450 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:36.838702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:42961)
0929 11:33:36.839506 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:36.839561 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:36.841505 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:36.842240 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:36.842248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:36.842897 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:36.843491 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:36.843510 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.844559 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.844563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:36.845900 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:36.845913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:36.846714 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:36.846721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:36.846944 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:36.847591 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:36.847615 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:36.850111 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:36.853027 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:36.853036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:36.853055 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:36.853490 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:36.854029 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:36.854034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:36.854039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:36.854257 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:36.855321 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:36.855334 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:36.856001 (+ 667us) client_negotiation.cc:769] Sending connection context
0929 11:33:36.856297 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 11:33:36.857091 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for oryx/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:33:36.860193 12625 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:36.860601 12625 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:36.861543 12625 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.195@KRBTEST.COM" created.
I20260929 11:33:36.863705 12302 ts_manager.cc:194] Registered new tserver with Master: b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:36.864537 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:42961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
I20260929 11:33:36.891093 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:37.212291 12630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:37.212589 12630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:37.212713 12630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:37.220106 12630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:37.220217 12630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:37.220340 12630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:37.230257 12630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:37.231455 12630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:37.232741 12630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:37.245440 12638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:37.245477 12635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:37.245514 12636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:37.246199 12630 server_base.cc:1034] running on GCE node
I20260929 11:33:37.246853 12630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:37.247797 12630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:37.248994 12630 hybrid_clock.cc:648] HybridClock initialized: now 1790681617248905 us; error 83 us; skew 500 ppm
I20260929 11:33:37.249298 12630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681617, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:37.252326 12630 init.cc:377] Logged in from keytab as oryx/127.6.9.195@KRBTEST.COM (short username oryx)
I20260929 11:33:37.253841 12630 webserver.cc:460] Webserver started at http://127.6.9.195:45393/ using document root <none> and password file <none>
I20260929 11:33:37.254522 12630 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:37.254623 12630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:37.254923 12630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:37.257830 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "36ef156af6f84be5957cefb702865298"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "6589924d49bd401097d379667c253459"
server_key_iv: "d2d1e5730e9206f35ba7efb97c06a81d"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.258597 12630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "36ef156af6f84be5957cefb702865298"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "6589924d49bd401097d379667c253459"
server_key_iv: "d2d1e5730e9206f35ba7efb97c06a81d"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.263731 12630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:37.267270 12645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:37.268713 12630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:37.268868 12630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "36ef156af6f84be5957cefb702865298"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "6589924d49bd401097d379667c253459"
server_key_iv: "d2d1e5730e9206f35ba7efb97c06a81d"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.269044 12630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:37.310747 12630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:37.315778 12630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:37.316033 12630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:37.316987 12630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:37.318277 12630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:37.318348 12630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:37.318436 12630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:37.318478 12630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:37.347503 12630 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:43307
I20260929 11:33:37.347524 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:43307 every 8 connection(s)
I20260929 11:33:37.349071 12630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:37.350615 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12630
I20260929 11:33:37.350760 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:37.351163 6183 external_mini_cluster.cc:1428] Setting key 4fa3b86763976a3abdf9534c560f1e73
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681617, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:37.368923 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.352082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:40315 (local address 127.6.9.254:46741)
0929 11:33:37.352276 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.352282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.353742 (+ 1460us) server_negotiation.cc:407] Connection header received
0929 11:33:37.355753 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.355758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.355850 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.355929 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.357339 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.358080 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.359580 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.359983 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.360077 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.363282 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.363287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.363290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.363323 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.366059 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.366866 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.366870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.366872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.366991 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.367381 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.367385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.367387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.367671 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.367787 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.368457 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.368628 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:33:37.369853 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.352736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:40315)
0929 11:33:37.353581 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.353633 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.355513 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.356096 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.356104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.356587 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.357192 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.357209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.358189 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.358192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.359430 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.359442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.360164 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.360169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.360362 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.360974 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.360996 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.363084 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.366182 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.366189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.366201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.366753 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.367071 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.367074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.367077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.367246 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.367775 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.367781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.368091 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.368313 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.368751 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":351,"threads_started":1}
I20260929 11:33:37.371112 12758 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:37.371413 12758 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:37.372166 12758 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
I20260929 11:33:37.373621 12302 ts_manager.cc:194] Registered new tserver with Master: 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:37.374400 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:40315
I20260929 11:33:37.378649 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:33:37.379626 12491 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.6.9.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.253.keytab.
Entry for principal oryx/127.6.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.253.keytab.
I20260929 11:33:37.422024 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:33433
--webserver_interface=127.6.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.253.keytab
--principal=oryx/127.6.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46741,127.6.9.253:33433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:37.740748 12763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:37.741076 12763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:37.741210 12763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:37.748355 12763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:37.748463 12763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:37.748539 12763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:37.748610 12763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:37.748687 12763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:37.758447 12763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46741,127.6.9.253:33433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.253.keytab
--principal=oryx/127.6.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:33433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
W20260929 11:33:37.758792 12763 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.9.254:46741,127.6.9.253:33433'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 11:33:37.759836 12763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:37.761157 12763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:37.772274 12769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:37.772398 12771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:37.772709 12768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:37.773094 12763 server_base.cc:1034] running on GCE node
I20260929 11:33:37.773857 12763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:37.775189 12763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:37.776386 12763 hybrid_clock.cc:648] HybridClock initialized: now 1790681617776320 us; error 47 us; skew 500 ppm
I20260929 11:33:37.776727 12763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681617, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:37.779718 12763 init.cc:377] Logged in from keytab as oryx/127.6.9.253@KRBTEST.COM (short username oryx)
I20260929 11:33:37.785552 12763 webserver.cc:460] Webserver started at http://127.6.9.253:34555/ using document root <none> and password file <none>
I20260929 11:33:37.786346 12763 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:37.786459 12763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:37.786763 12763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:37.789994 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/instance:
uuid: "4cee440cc6944cbe961cd9955592ed22"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "9da1add6e17f90f632a20ce56732e4b1"
server_key_iv: "ddaa9d5e7c0ee39716b87c3fcd70bd96"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.790781 12763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal/instance:
uuid: "4cee440cc6944cbe961cd9955592ed22"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "9da1add6e17f90f632a20ce56732e4b1"
server_key_iv: "ddaa9d5e7c0ee39716b87c3fcd70bd96"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.796041 12763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:33:37.799536 12778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:37.800932 12763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:37.801107 12763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "4cee440cc6944cbe961cd9955592ed22"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "9da1add6e17f90f632a20ce56732e4b1"
server_key_iv: "ddaa9d5e7c0ee39716b87c3fcd70bd96"
server_key_version: "encryptionkey@0"
I20260929 11:33:37.801303 12763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:37.817291 12763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:37.821552 12763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:37.821794 12763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:37.838994 12763 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.253:33433
I20260929 11:33:37.839015 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.253:33433 every 8 connection(s)
I20260929 11:33:37.840380 12763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
I20260929 11:33:37.844108 12830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:37.850867 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12763
I20260929 11:33:37.850582 12830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681617, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.253@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:37.867308 12625 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
I20260929 11:33:37.871469 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.853971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.6.9.254:46741)
0929 11:33:37.854176 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.854182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.854743 (+ 561us) server_negotiation.cc:407] Connection header received
0929 11:33:37.856723 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.856727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.856794 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.856891 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.858209 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.858956 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.860492 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.860948 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.861095 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.864793 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.864800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.864805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.864848 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.867794 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.868717 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.868723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.868727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.868850 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.869342 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.869347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.869351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.869770 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.869888 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.870871 (+ 983us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.871152 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:33:37.872825 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.853501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36522)
0929 11:33:37.854569 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.854629 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.856414 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.856986 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.856993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.857660 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.858038 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.858057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.859100 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.859104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.860349 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.860362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.861205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.861212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.861445 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.862150 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.862183 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.864563 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.867976 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.867998 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.868012 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.868560 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.868987 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.868993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.868996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.869217 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.869908 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.869917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.870389 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.870739 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.871285 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":42,"spinlock_wait_cycles":18304,"thread_start_us":664,"threads_started":1}
I20260929 11:33:37.872844 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.853393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.6.9.253:33433)
0929 11:33:37.854789 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.854797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.854839 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:33:37.856399 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.856427 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.856754 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.856928 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.858348 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.859578 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.860819 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.861409 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.861530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.865440 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.865449 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.865463 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.865518 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.868835 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.869683 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.869689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.869691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.869802 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.870176 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.870180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.870183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.870659 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.870921 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.871285 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.871526 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1179,"spinlock_wait_cycles":1408,"thread_start_us":918,"threads_started":1}
I20260929 11:33:37.874720 12830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681617, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.253@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:37.887290 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.875371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56544)
0929 11:33:37.875713 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.875727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.875831 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.876256 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.876261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.876613 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.876792 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.876801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.877720 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.877726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.878609 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.878620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.879272 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.879277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.879448 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.879995 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.880012 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.882101 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.884982 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.884995 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.885000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.885372 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.885715 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.885720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.885724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.885854 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.886344 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.886349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.886482 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.886601 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.886890 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260929 11:33:37.887324 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.875400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.6.9.253:33433)
0929 11:33:37.875536 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.875540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.875796 (+ 256us) server_negotiation.cc:407] Connection header received
0929 11:33:37.876088 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.876092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.876143 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.876218 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.876912 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.877602 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.878741 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.879134 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.879246 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.882300 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.882304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.882308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.882340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.884849 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.885480 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.885494 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.885497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.885626 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.885949 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.885954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.885957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.886241 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.886365 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.886698 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.887018 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":40}
I20260929 11:33:37.892323 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Bootstrap starting.
I20260929 11:33:37.895686 12830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:37.896785 12830 log.cc:824] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:37.899523 12830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: No bootstrap required, opened a new log
I20260929 11:33:37.906636 12830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } }
I20260929 11:33:37.907008 12830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:37.907114 12830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cee440cc6944cbe961cd9955592ed22, State: Initialized, Role: FOLLOWER
I20260929 11:33:37.907676 12830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } }
I20260929 11:33:37.908521 12836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } } }
I20260929 11:33:37.908838 12836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:37.909524 12830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:33:37.917724 12847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:33:37.917896 12847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:33:37.920378 12763 master_runner.cc:428] Detected that this master 4cee440cc6944cbe961cd9955592ed22 is joining an existing cluster
I20260929 11:33:37.920468 12763 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.9.253:33433
I20260929 11:33:37.940148 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.925552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.6.9.253:33433)
0929 11:33:37.925718 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.925723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.926358 (+ 635us) server_negotiation.cc:407] Connection header received
0929 11:33:37.926556 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.926560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.926620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.926686 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.928007 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.929047 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.930857 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.931561 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.931645 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.934125 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.934130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.934134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.934169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.937313 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.937907 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.937912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.937915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.938068 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.938406 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.938410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.938412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.938779 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.938902 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.939431 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.939736 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":34688}
I20260929 11:33:37.941350 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.925699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56548)
0929 11:33:37.926264 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.926281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.926447 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.926867 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.926871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.927547 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.927857 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.927870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.929250 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.929255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.930731 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.930745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.931969 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.931974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.932136 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.932627 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.932647 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.933944 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.937427 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.937432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.937436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.937805 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.938177 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.938180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.938184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.938317 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.938973 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.938977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.939197 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.940592 (+ 1395us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.940944 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":3,"thread_start_us":257,"threads_started":1}
I20260929 11:33:37.941855 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.924237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.6.9.254:46741)
0929 11:33:37.924414 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.924420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.925297 (+ 877us) server_negotiation.cc:407] Connection header received
0929 11:33:37.925495 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.925501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.925584 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.925695 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.927275 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.928397 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.930444 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.931097 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.931194 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.934246 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.934253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.934257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.934297 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.938199 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.938985 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.938991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.938995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.939159 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.939645 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.939651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.939654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.940012 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.940215 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.941066 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.941357 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:33:37.942840 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.924162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36528)
0929 11:33:37.925195 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.925216 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.925357 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.925935 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.925940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.926640 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.927086 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.927100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.928578 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.928583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.930281 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.930295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.931504 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.931511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.931710 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.932599 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.932626 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.934029 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.938356 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.938361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.938365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.938832 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.939313 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.939318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.939321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.939487 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.940475 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.940480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.940721 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.942097 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.942409 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":24,"thread_start_us":669,"threads_started":1}
WARNING: no policy specified for oryx/127.6.9.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.252@KRBTEST.COM" created.
Sep 29 11:33:37 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.254@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:37.967378 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.949897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56556)
0929 11:33:37.950616 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:37.950661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:37.950915 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:37.951380 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:37.951385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:37.952032 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:37.952235 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.952244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.953633 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.953639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.955172 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:37.955187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.956267 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.956272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:37.956464 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.957234 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:37.957255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:37.960110 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:37.964118 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.964136 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.964146 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.964827 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.965261 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:37.965266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:37.965269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:37.965455 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:37.966080 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:37.966089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:37.966323 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 11:33:37.966559 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 11:33:37.966925 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":330,"threads_started":1}
I20260929 11:33:37.968555 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:37.949912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.6.9.253:33433)
0929 11:33:37.950110 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:37.950117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:37.950757 (+ 640us) server_negotiation.cc:407] Connection header received
0929 11:33:37.951145 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:37.951152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:37.951243 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:37.951350 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:37.952371 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.953457 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.955351 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:37.956073 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:37.956201 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:37.960341 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:37.960349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:37.960354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:37.960403 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:37.963932 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.964957 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.964962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.964966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.965143 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:37.965590 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:37.965594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:37.965597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:37.965940 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:37.966113 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:37.967927 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0929 11:33:37.968202 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:33:37.970852 12302 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.9.253:33433: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4cee440cc6944cbe961cd9955592ed22"
member_type: NON_VOTER
last_known_addr {
host: "127.6.9.253"
port: 33433
}
attrs {
promote: true
}
}
dest_uuid: "715519d5829c48c48a4bcdfe3a49bfbf"
cas_config_opid_index: -1
I20260929 11:33:37.972350 12302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } }
I20260929 11:33:37.975762 12860 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } }
I20260929 11:33:37.977108 12805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.252.keytab.
Entry for principal oryx/127.6.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.252.keytab.
I20260929 11:33:37.979882 12860 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.9.253:33433
I20260929 11:33:37.980647 12763 master.cc:537] Master@127.6.9.253:33433 shutting down...
I20260929 11:33:37.980859 12805 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 11:33:37.980917 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:37129
--webserver_interface=127.6.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.252.keytab
--principal=oryx/127.6.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:33:37.981817 12860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } }
I20260929 11:33:37.982049 12860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:37.981909 12859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:33:37.985265 12289 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 11:33:37.996233 12763 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:37.996728 12763 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:37.996939 12763 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: stopping tablet replica
I20260929 11:33:38.005038 12763 master.cc:552] Master@127.6.9.253:33433 shutdown complete.
I20260929 11:33:38.005156 12763 master_runner.cc:305] Clearing existing system tablet
I20260929 11:33:38.005438 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.004955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56566)
0929 11:33:38.005119 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.005262 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:33433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:33:38.005587 12763 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:38.005689 12763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260929 11:33:38.005805 12289 consensus_peers.cc:487] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf -> Peer 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253:33433): Couldn't send request to peer 4cee440cc6944cbe961cd9955592ed22. Status: Network error: Client connection negotiation failed: client connection to 127.6.9.253:33433: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:33:38.007927 12763 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 11:33:38.009390 12865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:38.009681 12763 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:38.009984 12763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 11:33:38.010079 12763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "4cee440cc6944cbe961cd9955592ed22"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "9da1add6e17f90f632a20ce56732e4b1"
server_key_iv: "ddaa9d5e7c0ee39716b87c3fcd70bd96"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.013846 12763 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:38.018404 12763 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:33:38.018553 12763 log.cc:1196] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 11:33:38.018899 12763 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Deleting consensus metadata
I20260929 11:33:38.019124 12763 master_runner.cc:315] Copying system tablet from 127.6.9.254:46741
I20260929 11:33:38.024290 12763 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: tablet copy: Beginning tablet copy session from remote peer at address 127.6.9.254:46741
I20260929 11:33:38.035913 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.025043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.6.9.254:46741)
0929 11:33:38.025237 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.025244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.025687 (+ 443us) server_negotiation.cc:407] Connection header received
0929 11:33:38.025856 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.025862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.025946 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.026033 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.026839 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.027514 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.028716 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.029142 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.029251 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.030950 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.030955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.030958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.030990 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.033470 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.034060 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.034065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.034067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.034180 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.034523 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.034527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.034531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.034803 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.034934 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.035346 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.035584 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:33:38.035948 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.025020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36544)
0929 11:33:38.025583 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.025599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.025739 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.026085 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.026090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.026480 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.026703 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.026712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.027662 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.027667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.028560 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.028571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.029287 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.029293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.029459 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.030071 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.030088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.030746 (+ 658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.033595 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.033601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.033605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.033952 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.034285 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.034290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.034294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.034427 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.034914 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.034919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.035090 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.035225 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.035512 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":337,"threads_started":1}
I20260929 11:33:38.036971 12322 tablet_copy_service.cc:140] P 715519d5829c48c48a4bcdfe3a49bfbf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4cee440cc6944cbe961cd9955592ed22 ({username='oryx', principal='oryx/127.6.9.253@KRBTEST.COM'} at 127.0.0.1:36544)
I20260929 11:33:38.037222 12322 tablet_copy_service.cc:161] P 715519d5829c48c48a4bcdfe3a49bfbf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4cee440cc6944cbe961cd9955592ed22 at {username='oryx', principal='oryx/127.6.9.253@KRBTEST.COM'} at 127.0.0.1:36544: session id = 4cee440cc6944cbe961cd9955592ed22-00000000000000000000000000000000
I20260929 11:33:38.040388 12322 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:33:38.042328 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:33:38.046782 12763 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: tablet copy: Starting download of 0 data blocks...
I20260929 11:33:38.047070 12763 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: tablet copy: Starting download of 1 WAL segments...
I20260929 11:33:38.049428 12763 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:33:38.052685 12322 tablet_copy_service.cc:342] P 715519d5829c48c48a4bcdfe3a49bfbf: Request end of tablet copy session 4cee440cc6944cbe961cd9955592ed22-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.9.253@KRBTEST.COM'} at 127.0.0.1:36544
I20260929 11:33:38.052893 12322 tablet_copy_service.cc:433] P 715519d5829c48c48a4bcdfe3a49bfbf: ending tablet copy session 4cee440cc6944cbe961cd9955592ed22-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4cee440cc6944cbe961cd9955592ed22
I20260929 11:33:38.062320 12763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:38.064797 12872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:38.070211 12873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:38.077536 12763 server_base.cc:1034] running on GCE node
W20260929 11:33:38.077384 12875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:38.078095 12763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:38.078730 12763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:38.080068 12763 hybrid_clock.cc:648] HybridClock initialized: now 1790681618079988 us; error 38 us; skew 500 ppm
I20260929 11:33:38.080339 12763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:38.082336 12763 init.cc:377] Logged in from keytab as oryx/127.6.9.253@KRBTEST.COM (short username oryx)
I20260929 11:33:38.083774 12763 webserver.cc:460] Webserver started at http://127.6.9.253:39361/ using document root <none> and password file <none>
I20260929 11:33:38.084102 12763 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:38.084175 12763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:38.086915 12763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 11:33:38.088639 12882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:38.089128 12763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 11:33:38.089293 12763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "4cee440cc6944cbe961cd9955592ed22"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "9da1add6e17f90f632a20ce56732e4b1"
server_key_iv: "ddaa9d5e7c0ee39716b87c3fcd70bd96"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.089432 12763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:38.099448 12763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:38.102759 12763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:38.124528 12763 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.253:33433
I20260929 11:33:38.124565 12933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.253:33433 every 8 connection(s)
I20260929 11:33:38.125443 12763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
I20260929 11:33:38.130944 12934 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:33:38.134166 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Bootstrap starting.
I20260929 11:33:38.147035 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:33:38.147517 12934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: Bootstrap complete.
I20260929 11:33:38.148361 12934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } }
I20260929 11:33:38.148592 12934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4cee440cc6944cbe961cd9955592ed22, State: Initialized, Role: LEARNER
I20260929 11:33:38.148865 12934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } }
I20260929 11:33:38.149624 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } }
I20260929 11:33:38.149950 12934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:38.150135 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:33:38.158069 12948 catalog_manager.cc:1259] Loaded cluster ID: e87b41d2600742d9a2565ab6f49a0807
I20260929 11:33:38.158166 12948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: loading cluster ID for follower catalog manager: success
I20260929 11:33:38.160463 12948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: acquiring CA information for follower catalog manager: success
I20260929 11:33:38.162228 12948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:33:38.374043 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:38.374359 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:38.374486 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 11:33:38.376811 12758 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
W20260929 11:33:38.382087 12863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:38.382201 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:38.382278 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:38.382351 12863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:38.382407 12863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:38.392959 12863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.252.keytab
--principal=oryx/127.6.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:38.394395 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:38.395716 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:38.406805 12953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:38.406728 12954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:38.406765 12956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:38.407740 12863 server_base.cc:1034] running on GCE node
I20260929 11:33:38.408531 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:38.410043 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:38.411266 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790681618411217 us; error 47 us; skew 500 ppm
I20260929 11:33:38.411613 12863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:38.414695 12863 init.cc:377] Logged in from keytab as oryx/127.6.9.252@KRBTEST.COM (short username oryx)
I20260929 11:33:38.416314 12863 webserver.cc:460] Webserver started at http://127.6.9.252:46787/ using document root <none> and password file <none>
I20260929 11:33:38.417064 12863 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:38.417181 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:38.417459 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:38.420513 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/instance:
uuid: "91dee2623f154dd1b596f7625f8641ee"
format_stamp: "Formatted at 2026-09-29 11:33:38 on dist-test-slave-c0l9"
server_key: "fdcad022b702fa4fb51606b8919f400d"
server_key_iv: "ac1afd136cff3625c02a093b570ec11a"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.421368 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal/instance:
uuid: "91dee2623f154dd1b596f7625f8641ee"
format_stamp: "Formatted at 2026-09-29 11:33:38 on dist-test-slave-c0l9"
server_key: "fdcad022b702fa4fb51606b8919f400d"
server_key_iv: "ac1afd136cff3625c02a093b570ec11a"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.426594 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:38.430095 12963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:38.431577 12863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 11:33:38.431761 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "91dee2623f154dd1b596f7625f8641ee"
format_stamp: "Formatted at 2026-09-29 11:33:38 on dist-test-slave-c0l9"
server_key: "fdcad022b702fa4fb51606b8919f400d"
server_key_iv: "ac1afd136cff3625c02a093b570ec11a"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.431978 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:38.468106 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:38.472700 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:38.472987 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:38.490737 12863 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.252:37129
I20260929 11:33:38.490780 13014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.252:37129 every 8 connection(s)
I20260929 11:33:38.492106 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
I20260929 11:33:38.495661 13015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:38.498215 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 12863
I20260929 11:33:38.503473 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.252@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:38.520063 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.506077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.6.9.254:46741)
0929 11:33:38.506222 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.506226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.506924 (+ 698us) server_negotiation.cc:407] Connection header received
0929 11:33:38.507110 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.507114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.507177 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.507258 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.508268 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.509000 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.510460 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.510860 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.510964 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.514270 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.514278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.514283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.514326 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.517156 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.517948 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.517953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.517956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.518113 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.518422 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.518426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.518430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.518792 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.518936 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.519490 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.519734 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:33:38.520840 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.506095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36554)
0929 11:33:38.506788 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.506826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.506987 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.507333 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.507340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.507894 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.508130 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.508143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.509137 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.509142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.510328 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.510340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.511072 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.511080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.511217 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.511796 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.511818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.514068 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.517283 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.517292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.517298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.517842 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.518204 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.518207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.518209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.518331 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.518920 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.518927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.519133 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.519374 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.519692 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":46,"thread_start_us":362,"threads_started":1}
I20260929 11:33:38.520857 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.500491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.6.9.252:37129)
0929 11:33:38.501266 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.501281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.501324 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:33:38.502948 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.502971 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.503371 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.503753 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.505271 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.507208 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.508410 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.509045 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.509254 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.513241 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.513249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.513261 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.513295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.516532 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.517251 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.517260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.517269 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.517440 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.517839 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.517842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.517844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.518527 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.518808 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.519132 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.519495 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":570,"thread_start_us":334,"threads_started":1}
I20260929 11:33:38.522274 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.252@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:38.534686 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.522887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56580)
0929 11:33:38.523059 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.523078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.523172 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.523869 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.523872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.524186 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.524320 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.524328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.525747 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.525751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.526684 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.526695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.527323 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.527327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.527486 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.527988 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.528007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.529984 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.532728 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.532732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.532735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.533122 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.533439 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.533442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.533444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.533574 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.533985 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.533988 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.534111 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.534213 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.534428 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 11:33:38.534715 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.523052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.6.9.253:33433)
0929 11:33:38.523617 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.523624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.523639 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:33:38.523686 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.523689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.523746 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.523830 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.524483 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.525611 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.526800 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.527210 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.527298 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.530177 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.530182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.530186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.530219 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.532557 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.533219 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.533223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.533225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.533360 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.533647 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.533650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.533652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.533905 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.534021 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.534280 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.534446 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260929 11:33:38.535795 13015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.252@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
I20260929 11:33:38.547014 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.536507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33454)
0929 11:33:38.536724 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.536742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.536825 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.537057 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.537060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.537362 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.537497 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.537516 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.538403 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.538407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.539242 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.539253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.539870 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.539873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.540057 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.540535 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.540551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.542414 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.545027 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.545030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.545032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.545347 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.545674 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.545678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.545680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.545831 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.546292 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.546295 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.546412 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.546522 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.546716 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 11:33:38.547057 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.536502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.6.9.252:37129)
0929 11:33:38.536761 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.536765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.536781 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:33:38.536902 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.536905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.536957 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.537028 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.537610 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.538298 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.539347 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.539749 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.539854 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.542590 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.542595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.542598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.542627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.544919 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.545437 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.545441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.545443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.545549 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.545931 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.545935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.545937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.546197 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.546298 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.546585 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.546827 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":133}
I20260929 11:33:38.551972 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Bootstrap starting.
I20260929 11:33:38.555141 13015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:38.556137 13015 log.cc:824] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:38.558732 13015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: No bootstrap required, opened a new log
I20260929 11:33:38.565644 13015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } }
I20260929 11:33:38.565989 13015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:38.566104 13015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91dee2623f154dd1b596f7625f8641ee, State: Initialized, Role: FOLLOWER
I20260929 11:33:38.566627 13015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } }
I20260929 11:33:38.567415 13022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } } }
I20260929 11:33:38.567701 13022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:38.568457 13015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:38.575588 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12363
W20260929 11:33:38.577353 13033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:33:38.577495 13033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:33:38.582262 12863 master_runner.cc:428] Detected that this master 91dee2623f154dd1b596f7625f8641ee is joining an existing cluster
I20260929 11:33:38.582386 12863 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.9.252:37129
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.193@KRBTEST.COM
I20260929 11:33:38.601154 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.576887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.6.9.193:42045)
0929 11:33:38.578039 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.578050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.578087 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:33:38.578157 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.578165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.578414 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.578575 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.580123 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.581713 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.584219 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.585282 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.591758 (+ 6476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.592104 (+ 346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.592118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.592140 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.592176 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.596378 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.597448 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.597459 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.597463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.597632 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.598122 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.598127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.598130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.598623 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.600191 (+ 1568us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.600376 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.600698 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":823,"threads_started":1}
I20260929 11:33:38.603075 12486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:38.623876 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.605205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.6.9.253:33433)
0929 11:33:38.605661 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.605667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.615995 (+ 10328us) server_negotiation.cc:407] Connection header received
0929 11:33:38.616503 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.616510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.616617 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.616724 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:38.617417 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.618562 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.621235 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.622556 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.622661 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.622887 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.623447 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.623610 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260929 11:33:38.624732 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.605061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56582)
0929 11:33:38.615850 (+ 10789us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.615874 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.616308 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.616971 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.616976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.617011 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:38.617252 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.617264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.618908 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.618914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.621038 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.621055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.623037 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.623043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.623233 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.623298 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.624212 (+ 914us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.624364 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10634,"thread_start_us":10951,"threads_started":1}
I20260929 11:33:38.625905 12909 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 11:33:38.626546 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 715519d5829c48c48a4bcdfe3a49bfbf. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } }
I20260929 11:33:38.626827 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:33:38.647384 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.630743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33460)
0929 11:33:38.631449 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.631470 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.631667 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.632205 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.632211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.632950 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.633328 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.633341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.634823 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.634828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.636559 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.636574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.637845 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.637851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.638022 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.638753 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.638776 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.640199 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.644564 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.644568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.644573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.645064 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.645458 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.645462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.645466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.645620 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.646319 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.646323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.646478 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.646612 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.646898 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":611,"threads_started":1}
I20260929 11:33:38.648294 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.631193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.6.9.252:37129)
0929 11:33:38.631350 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.631359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.631553 (+ 194us) server_negotiation.cc:407] Connection header received
0929 11:33:38.631830 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.631836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.631904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.632001 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.633485 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.634605 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.636728 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.637419 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.637523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.640365 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.640372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.640377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.640409 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.644429 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.645184 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.645189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.645192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.645340 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.645745 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.645749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.645753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.646141 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.646275 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.647697 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.647921 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:33:38.661410 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.630594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.6.9.254:46741)
0929 11:33:38.630801 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.630806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.638763 (+ 7957us) server_negotiation.cc:407] Connection header received
0929 11:33:38.640239 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.640245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.640327 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.640391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.644336 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.645494 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.650650 (+ 5156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.651312 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.651431 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.654371 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.654379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.654384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.654425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.658017 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.658906 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.658912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.658915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.659082 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.659539 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.659544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.659547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.659950 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.660078 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.660793 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.661036 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:33:38.661489 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.639407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.6.9.253:33433)
0929 11:33:38.639609 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.639614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.641754 (+ 2140us) server_negotiation.cc:407] Connection header received
0929 11:33:38.641967 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.641973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.642047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.642109 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.646150 (+ 4041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.647243 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.651142 (+ 3899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.651820 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.651915 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.654457 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.654464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.654469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.654517 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.658260 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.659093 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.659099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.659102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.659245 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.659737 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.659743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.659746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.660086 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.660205 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.660865 (+ 660us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.661102 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:33:38.662587 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.631889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36562)
0929 11:33:38.638648 (+ 6759us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.638668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.639958 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.640607 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.640612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.643208 (+ 2596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.643611 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.643623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.648814 (+ 5191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.648819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.650465 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.650484 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.651760 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.651767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.651945 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.652721 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.652744 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.654108 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.658182 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.658187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.658191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.658730 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.659223 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.659227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.659230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.659407 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.660154 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.660158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.660383 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.661666 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.662048 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6598,"mutex_wait_us":2,"spinlock_wait_cycles":25344,"thread_start_us":6891,"threads_started":1}
I20260929 11:33:38.662917 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.639735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:56598)
0929 11:33:38.641655 (+ 1920us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.641674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.641841 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.642294 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.642299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.645617 (+ 3318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.645974 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.645988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.647411 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.647416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.650977 (+ 3561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.650991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.652196 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.652202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.652395 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.653027 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.653049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.654249 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.658444 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.658449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.658453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.658963 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.659379 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.659402 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.659406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.659621 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.660277 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.660282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.660544 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.661727 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.662241 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1796,"mutex_wait_us":177,"thread_start_us":2589,"threads_started":1}
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.254@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
I20260929 11:33:38.682987 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.667790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33476)
0929 11:33:38.668068 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.668086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.668192 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.668596 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.668601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.669111 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.669296 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.669308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.670654 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.670659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.672068 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.672082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.673107 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.673113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.673285 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.673865 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.673888 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.676425 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.680245 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.680251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.680255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.680792 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.681201 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.681207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.681211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.681357 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.681956 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.681962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.682148 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.682297 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.682615 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 11:33:38.682981 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.667863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.6.9.252:37129)
0929 11:33:38.668021 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.668027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.668136 (+ 109us) server_negotiation.cc:407] Connection header received
0929 11:33:38.668329 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.668334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.668438 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.668523 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.669429 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.670526 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.672210 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.672949 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.673060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.676650 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.676657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.676685 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.676722 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.680102 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.680940 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.680946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.680950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.681094 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.681492 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.681498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.681501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.681836 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.682014 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.682397 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.682613 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:33:38.684940 12302 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.9.252:37129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91dee2623f154dd1b596f7625f8641ee"
member_type: NON_VOTER
last_known_addr {
host: "127.6.9.252"
port: 37129
}
attrs {
promote: true
}
}
dest_uuid: "715519d5829c48c48a4bcdfe3a49bfbf"
cas_config_opid_index: 5
I20260929 11:33:38.685995 12302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } }
I20260929 11:33:38.687740 13036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91dee2623f154dd1b596f7625f8641ee (127.6.9.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.688423 12909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 11:33:38.689998 12990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:38.693495 12990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 11:33:38.701485 13046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 11:33:38.709040 13036 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.9.252:37129
I20260929 11:33:38.709137 13047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:38.709743 12863 master.cc:537] Master@127.6.9.252:37129 shutting down...
I20260929 11:33:38.710681 13048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.711056 13048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
W20260929 11:33:38.719516 12286 consensus_peers.cc:487] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf -> Peer 91dee2623f154dd1b596f7625f8641ee (127.6.9.252:37129): Couldn't send request to peer 91dee2623f154dd1b596f7625f8641ee. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 11:33:38.719759 12863 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:38.720160 12863 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:38.720343 12863 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: stopping tablet replica
I20260929 11:33:38.720080 12909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91dee2623f154dd1b596f7625f8641ee (127.6.9.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.726255 12863 master.cc:552] Master@127.6.9.252:37129 shutdown complete.
I20260929 11:33:38.726343 12863 master_runner.cc:305] Clearing existing system tablet
I20260929 11:33:38.726728 12863 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:38.726838 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:38.730209 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: NON_VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: true } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.730453 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: LEARNER
I20260929 11:33:38.730942 12863 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:38.731420 13047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: attempting to promote NON_VOTER 4cee440cc6944cbe961cd9955592ed22 to VOTER
I20260929 11:33:38.732498 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:38.732299 13047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } }
I20260929 11:33:38.732966 12863 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:38.733215 12863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 11:33:38.733309 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "91dee2623f154dd1b596f7625f8641ee"
format_stamp: "Formatted at 2026-09-29 11:33:38 on dist-test-slave-c0l9"
server_key: "fdcad022b702fa4fb51606b8919f400d"
server_key_iv: "ac1afd136cff3625c02a093b570ec11a"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.734381 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.733915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33492)
0929 11:33:38.734063 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.734179 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 11:33:38.735874 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.735427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33504)
0929 11:33:38.735651 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.735738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:37129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 11:33:38.735877 12909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 LEARNER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 11:33:38.736150 12286 consensus_peers.cc:487] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf -> Peer 91dee2623f154dd1b596f7625f8641ee (127.6.9.252:37129): Couldn't send request to peer 91dee2623f154dd1b596f7625f8641ee. Status: Network error: Client connection negotiation failed: client connection to 127.6.9.252:37129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:33:38.736469 13048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 11:33:38.737406 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12363
I20260929 11:33:38.738889 13047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.740863 12863 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:38.742220 13048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.742470 13048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:38.742754 13048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.742973 13048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:38.742645 12909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.746536 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.746779 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:38.747008 12863 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:33:38.747159 12863 log.cc:1196] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 11:33:38.747562 12863 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Deleting consensus metadata
I20260929 11:33:38.747777 12863 master_runner.cc:315] Copying system tablet from 127.6.9.254:46741
I20260929 11:33:38.759174 12863 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.9.254:46741
I20260929 11:33:38.759478 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:42045
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:46741
--webserver_port=39633
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:33:38.778821 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.759874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:36574)
0929 11:33:38.760366 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:38.760386 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:38.760493 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:38.765340 (+ 4847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:38.765345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:38.765749 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:38.765964 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.765977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.767521 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.767527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.768910 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:38.768927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.769985 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.769990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:38.770152 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.770831 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:38.770853 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:38.771882 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:38.775839 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.775845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.775850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.776351 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.776831 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:38.776837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:38.776840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:38.777037 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:38.777689 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:38.777694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:38.777899 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 11:33:38.778063 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:33:38.778365 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":280,"threads_started":1}
I20260929 11:33:38.778855 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:38.764875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.6.9.254:46741)
0929 11:33:38.765053 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:38.765057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:38.765074 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:33:38.765128 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:38.765132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:38.765193 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:38.765285 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:38.766161 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.767344 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.769092 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:38.769814 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:38.769968 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:38.772088 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:38.772097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:38.772103 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:38.772146 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:38.775667 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.776481 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.776487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.776491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.776706 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:38.777162 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:38.777168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:38.777171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:38.777545 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:38.777700 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:38.778176 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 11:33:38.778445 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:33:38.779882 12322 tablet_copy_service.cc:140] P 715519d5829c48c48a4bcdfe3a49bfbf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91dee2623f154dd1b596f7625f8641ee ({username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:36574)
I20260929 11:33:38.780071 12322 tablet_copy_service.cc:161] P 715519d5829c48c48a4bcdfe3a49bfbf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91dee2623f154dd1b596f7625f8641ee at {username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:36574: session id = 91dee2623f154dd1b596f7625f8641ee-00000000000000000000000000000000
I20260929 11:33:38.784065 12322 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:33:38.786162 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:33:38.792944 12863 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: tablet copy: Starting download of 0 data blocks...
I20260929 11:33:38.793233 12863 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: tablet copy: Starting download of 1 WAL segments...
I20260929 11:33:38.795356 12863 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:33:38.799407 12322 tablet_copy_service.cc:342] P 715519d5829c48c48a4bcdfe3a49bfbf: Request end of tablet copy session 91dee2623f154dd1b596f7625f8641ee-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:36574
I20260929 11:33:38.799530 12322 tablet_copy_service.cc:433] P 715519d5829c48c48a4bcdfe3a49bfbf: ending tablet copy session 91dee2623f154dd1b596f7625f8641ee-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91dee2623f154dd1b596f7625f8641ee
I20260929 11:33:38.809448 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:38.811766 13061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:38.816751 13062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:38.823593 13064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:38.824476 12863 server_base.cc:1034] running on GCE node
I20260929 11:33:38.824821 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:38.825503 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:38.826682 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790681618826650 us; error 27 us; skew 500 ppm
I20260929 11:33:38.826916 12863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:38 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681618, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:38.828769 12863 init.cc:377] Logged in from keytab as oryx/127.6.9.252@KRBTEST.COM (short username oryx)
I20260929 11:33:38.836398 12863 webserver.cc:460] Webserver started at http://127.6.9.252:41551/ using document root <none> and password file <none>
I20260929 11:33:38.836710 12863 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:38.836820 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:38.839162 12863 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:38.840838 13071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:38.841315 12863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 11:33:38.841497 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "91dee2623f154dd1b596f7625f8641ee"
format_stamp: "Formatted at 2026-09-29 11:33:38 on dist-test-slave-c0l9"
server_key: "fdcad022b702fa4fb51606b8919f400d"
server_key_iv: "ac1afd136cff3625c02a093b570ec11a"
server_key_version: "encryptionkey@0"
I20260929 11:33:38.841666 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:38.863785 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:38.867131 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:38.891719 12863 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.252:37129
I20260929 11:33:38.891752 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.252:37129 every 8 connection(s)
I20260929 11:33:38.892556 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
I20260929 11:33:38.898089 13123 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:33:38.902275 13123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Bootstrap starting.
I20260929 11:33:38.918741 13123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:33:38.919193 13123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Bootstrap complete.
I20260929 11:33:38.920077 13123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } }
I20260929 11:33:38.920344 13123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91dee2623f154dd1b596f7625f8641ee, State: Initialized, Role: LEARNER
I20260929 11:33:38.920598 13123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } }
I20260929 11:33:38.921413 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:38.921679 13123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:38.921839 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: LEARNER
I20260929 11:33:38.936156 13137 catalog_manager.cc:1259] Loaded cluster ID: e87b41d2600742d9a2565ab6f49a0807
I20260929 11:33:38.936415 13137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: loading cluster ID for follower catalog manager: success
I20260929 11:33:38.939225 13137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: acquiring CA information for follower catalog manager: success
I20260929 11:33:38.941459 13137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:33:39.170400 13060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:39.170696 13060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:39.170853 13060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:39.178866 13060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:39.178992 13060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:39.179152 13060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:39.189826 13060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:42045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=39633
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:39.191248 13060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:39.192652 13060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:39.205643 13145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:39.205665 13143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:39.206194 13142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:39.206413 13060 server_base.cc:1034] running on GCE node
I20260929 11:33:39.206979 13060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:39.207873 13060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:39.209079 13060 hybrid_clock.cc:648] HybridClock initialized: now 1790681619209026 us; error 46 us; skew 500 ppm
I20260929 11:33:39.209385 13060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:39.212425 13060 init.cc:377] Logged in from keytab as oryx/127.6.9.193@KRBTEST.COM (short username oryx)
I20260929 11:33:39.213913 13060 webserver.cc:460] Webserver started at http://127.6.9.193:39633/ using document root <none> and password file <none>
I20260929 11:33:39.214607 13060 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:39.214738 13060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:39.220486 13060 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:33:39.223668 13152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.225421 13060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:39.225600 13060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "38d8dae75ba16744fd9592064a4925f8"
server_key_iv: "43eaf68535ca38f0775a9d0af5130cda"
server_key_version: "encryptionkey@0"
I20260929 11:33:39.226624 13060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:39.256301 13060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:39.261035 13060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:39.261330 13060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:39.262357 13060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:39.263768 13060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:39.263882 13060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.263998 13060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:39.264068 13060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.294775 13060 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:42045
I20260929 11:33:39.294839 13264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:42045 every 8 connection(s)
I20260929 11:33:39.296156 13060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:39.305231 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13060
I20260929 11:33:39.305646 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12497
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.194@KRBTEST.COM
I20260929 11:33:39.346356 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.299016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:51819 (local address 127.6.9.252:37129)
0929 11:33:39.299812 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.299817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.307209 (+ 7392us) server_negotiation.cc:407] Connection header received
0929 11:33:39.309819 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.309825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.309916 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.310020 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.315885 (+ 5865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.317402 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.322514 (+ 5112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.323296 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.323403 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.338523 (+ 15120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.338530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.338535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.338578 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.342160 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343063 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.343069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.343073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.343278 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343850 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.343855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.343859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.344190 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.344332 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.345745 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.345966 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":579,"threads_started":1}
I20260929 11:33:39.346381 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.307134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.6.9.194:33983)
0929 11:33:39.311789 (+ 4655us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.311798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.321546 (+ 9748us) server_negotiation.cc:407] Connection header received
0929 11:33:39.321843 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.321857 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.322200 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.322380 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.324878 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.326640 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.328761 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.329749 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.329860 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.336585 (+ 6725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.336610 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.336633 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.336704 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.342073 (+ 5369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343204 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.343213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.343216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.343366 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343802 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.343807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.343810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.344249 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.344525 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.345698 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.345975 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4480,"thread_start_us":4360,"threads_started":1}
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:39.347556 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.307573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:57547 (local address 127.6.9.254:46741)
0929 11:33:39.312130 (+ 4557us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.312136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.312154 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:39.312212 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.312217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.312288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.312395 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.314682 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.315700 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.324261 (+ 8561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.325075 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.325206 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.333566 (+ 8360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.333574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.333579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.333629 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.337391 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.339424 (+ 2033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.339430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.339433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.340464 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.344175 (+ 3711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.344180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.344183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.344556 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.344736 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.346893 (+ 2157us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.347172 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4437,"thread_start_us":4341,"threads_started":1}
I20260929 11:33:39.348697 12620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:39.348973 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.307001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:57547)
0929 11:33:39.311131 (+ 4130us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.311157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.311272 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.313682 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.313687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.314218 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.314490 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.314501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.318725 (+ 4224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.318730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.324070 (+ 5340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.324093 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.326256 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.326262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.326452 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.329340 (+ 2888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.329365 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.332353 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.337622 (+ 5269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.337633 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.337655 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.338281 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.343864 (+ 5583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.343871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.344019 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.344802 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.344807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.345151 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.345485 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.346051 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3973,"spinlock_wait_cycles":19840,"thread_start_us":5489,"threads_started":1}
I20260929 11:33:39.350744 13265 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:39.351117 13265 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.352068 13265 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
I20260929 11:33:39.354144 12302 ts_manager.cc:194] Re-registered known tserver with Master: 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045)
I20260929 11:33:39.354272 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.299601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.6.9.193:51819)
0929 11:33:39.306732 (+ 7131us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.306820 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.309460 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.312748 (+ 3288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.312752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.313228 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.313496 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.313509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.317617 (+ 4108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.317621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.322330 (+ 4709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.322349 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.323787 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.323795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.325709 (+ 1914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.326854 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.326887 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.333258 (+ 6371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.342331 (+ 9073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.342336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.342341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.342914 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.343432 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.343437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.343441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.343738 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.344412 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.344421 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.345157 (+ 736us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.353409 (+ 8252us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.353784 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6692,"thread_start_us":324,"threads_started":1}
I20260929 11:33:39.355296 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:57547
I20260929 11:33:39.356750 13266 heartbeater.cc:347] Connected to a master server at 127.6.9.252:37129
I20260929 11:33:39.356889 13266 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.357462 13266 heartbeater.cc:511] Master 127.6.9.252:37129 requested a full tablet report, sending...
I20260929 11:33:39.359357 13088 ts_manager.cc:194] Registered new tserver with Master: 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045)
I20260929 11:33:39.360526 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.319944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:33516)
0929 11:33:39.349437 (+ 29493us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.349457 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.350080 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.350759 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.350764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.350794 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:39.351093 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.351107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.354477 (+ 3370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.354484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.357864 (+ 3380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.357881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.359793 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.359800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.359983 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.360017 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.360105 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.360248 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29357,"mutex_wait_us":1456,"thread_start_us":31338,"threads_started":1}
I20260929 11:33:39.361589 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.298971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:52487 (local address 127.6.9.253:33433)
0929 11:33:39.299522 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.299529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.306948 (+ 7419us) server_negotiation.cc:407] Connection header received
0929 11:33:39.309856 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.309861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.309952 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.310023 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.314191 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.315335 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.338762 (+ 23427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.339424 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.339530 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.351412 (+ 11882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.351420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.351425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.351470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.354989 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.359073 (+ 4084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.359094 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.359099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.359294 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.359793 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.359799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.359803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.360197 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.360337 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.360972 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.361229 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":250,"threads_started":1}
I20260929 11:33:39.361662 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.300030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.6.9.193:52487)
0929 11:33:39.306767 (+ 6737us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.306816 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.309465 (+ 2649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.310605 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.310621 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.311713 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.312679 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.312710 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.315596 (+ 2886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.315601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.338548 (+ 22947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.338562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.339888 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.339895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.340071 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.345223 (+ 5152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.345245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.350538 (+ 5293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.355428 (+ 4890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.355434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.355439 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.358894 (+ 3455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.359459 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.359465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.359469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.359660 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.360426 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.360432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.360597 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.360841 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.361211 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6600,"spinlock_wait_cycles":26496,"thread_start_us":6392,"threads_started":1}
I20260929 11:33:39.362301 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.320216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.6.9.252:37129)
0929 11:33:39.346612 (+ 26396us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.346617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.350174 (+ 3557us) server_negotiation.cc:407] Connection header received
0929 11:33:39.350245 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.350250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.350346 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.350463 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:39.351227 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.352587 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.358046 (+ 5459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.359299 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.359450 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.359667 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.361879 (+ 2212us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.362042 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26276,"thread_start_us":28743,"threads_started":1}
I20260929 11:33:39.362910 13267 heartbeater.cc:347] Connected to a master server at 127.6.9.253:33433
I20260929 11:33:39.363061 13267 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.363278 13098 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 11:33:39.363540 13267 heartbeater.cc:511] Master 127.6.9.253:33433 requested a full tablet report, sending...
I20260929 11:33:39.363829 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 715519d5829c48c48a4bcdfe3a49bfbf. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:39.364096 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: LEARNER
I20260929 11:33:39.365309 12899 ts_manager.cc:194] Registered new tserver with Master: 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045)
I20260929 11:33:39.447566 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12497
I20260929 11:33:39.460611 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:33983
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:46741
--webserver_port=40031
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:33:39.757287 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: NON_VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: true } } }
I20260929 11:33:39.757663 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:39.758968 13274 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: attempting to promote NON_VOTER 91dee2623f154dd1b596f7625f8641ee to VOTER
I20260929 11:33:39.759518 13274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:39.761679 12909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:33:39.761716 13098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 LEARNER]: Refusing update from remote peer 715519d5829c48c48a4bcdfe3a49bfbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:33:39.762298 13283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:33:39.762732 13274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:33:39.771289 13274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91dee2623f154dd1b596f7625f8641ee (127.6.9.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.772415 12909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91dee2623f154dd1b596f7625f8641ee (127.6.9.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.772706 13098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91dee2623f154dd1b596f7625f8641ee (127.6.9.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.775444 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.775763 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:39.776072 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.776279 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:39.776618 13273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.776865 13273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:39.777626 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.777904 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:39.779675 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:39.780082 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 11:33:39.787873 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:39.788192 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:39.788316 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:39.795837 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:39.795967 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:39.796116 13282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:39.806456 13282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:33983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=40031
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:39.807847 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:39.809211 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:39.822885 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:39.823055 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:39.823446 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:39.823838 13282 server_base.cc:1034] running on GCE node
I20260929 11:33:39.824538 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:39.825661 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:39.826865 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790681619826827 us; error 44 us; skew 500 ppm
I20260929 11:33:39.827186 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:39.830446 13282 init.cc:377] Logged in from keytab as oryx/127.6.9.194@KRBTEST.COM (short username oryx)
I20260929 11:33:39.831868 13282 webserver.cc:460] Webserver started at http://127.6.9.194:40031/ using document root <none> and password file <none>
I20260929 11:33:39.832561 13282 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:39.832706 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:39.838085 13282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 11:33:39.840917 13303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.842294 13282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:33:39.842443 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "b98d47edc02a42d59ef41d5553f37fb1"
format_stamp: "Formatted at 2026-09-29 11:33:36 on dist-test-slave-c0l9"
server_key: "7ecbc84dcd249e818eb5d80f816d0f08"
server_key_iv: "e1bc6825d6ff9610630a3bf909f0068b"
server_key_version: "encryptionkey@0"
I20260929 11:33:39.843425 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:39.868760 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:39.873201 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:39.873473 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:39.874469 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:39.875953 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:39.876044 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.876147 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:39.876223 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:39.907703 13282 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:33983
I20260929 11:33:39.907759 13415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:33983 every 8 connection(s)
I20260929 11:33:39.909178 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:39.911695 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13282
I20260929 11:33:39.911994 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12630
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.195@KRBTEST.COM
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
Sep 29 11:33:39 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681619, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:39.965680 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.913293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:45627 (local address 127.6.9.253:33433)
0929 11:33:39.940779 (+ 27486us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.940785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.940805 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:33:39.943732 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.943739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.943838 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.943942 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.946628 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.947890 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.950849 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.951569 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.951687 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.958509 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.958516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.958520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.958574 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.961738 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.962721 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.962727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.962730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.962872 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963405 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.963409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.963412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.963791 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.963938 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.965040 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.965293 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27340,"thread_start_us":27570,"threads_started":1}
I20260929 11:33:39.965731 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.912137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:49693 (local address 127.6.9.252:37129)
0929 11:33:39.939476 (+ 27339us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.939483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.943609 (+ 4126us) server_negotiation.cc:407] Connection header received
0929 11:33:39.947867 (+ 4258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.947874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.947961 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.948070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.949650 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.950879 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.954040 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.954695 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.954805 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.958999 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.959007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.959011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.959054 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.962248 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963023 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.963029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.963032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.963158 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963564 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.963569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.963572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.963912 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.964022 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.965054 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.965287 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27201,"thread_start_us":27062,"threads_started":1}
I20260929 11:33:39.967046 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.913293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.6.9.195:43307)
0929 11:33:39.938005 (+ 24712us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.938016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.938050 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:33:39.938127 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.938137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.938449 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.938642 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.940632 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.942872 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.944980 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.945911 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.946050 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.957376 (+ 11326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.957397 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.957418 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.957462 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.962687 (+ 5225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963651 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.963661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.963664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.963832 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.964329 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.964334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.964338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.964801 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.965099 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.966358 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.966622 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24498,"thread_start_us":26199,"threads_started":1}
I20260929 11:33:39.968106 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.912704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.6.9.194:45627)
0929 11:33:39.940362 (+ 27658us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.940425 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.943161 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.944448 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.944463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.945416 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.946339 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.946369 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.948169 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.948175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.950637 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.950657 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.952106 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.952114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.953025 (+ 911us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.954224 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.954258 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.958228 (+ 3970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.961971 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.961981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.962002 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.962579 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.963011 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.963018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.963284 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.964030 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.964038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.964593 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.966326 (+ 1733us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.966633 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27301,"mutex_wait_us":507,"spinlock_wait_cycles":40064,"thread_start_us":28118,"threads_started":1}
I20260929 11:33:39.968127 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.912703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.6.9.194:49693)
0929 11:33:39.941326 (+ 28623us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.941345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.947714 (+ 6369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.948313 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.948318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.949101 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.949436 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.949447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.952021 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.952031 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.953887 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.953904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.955069 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.955076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.955242 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.955845 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.955868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.958765 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.962395 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.962400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.962403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.962902 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.963298 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.963302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.963305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.963452 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.964122 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.964126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.964579 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.965926 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.966505 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28440,"mutex_wait_us":720,"thread_start_us":26464,"threads_started":1}
I20260929 11:33:39.968502 12753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:39.969732 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.941518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:50647)
0929 11:33:39.952902 (+ 11384us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:39.952923 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:39.953043 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:39.953554 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:39.953559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:39.954221 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:39.954566 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.954577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.955901 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.955906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.957339 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:39.957353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.958537 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.958544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:39.958732 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.959241 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:39.959265 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:39.961778 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:39.965950 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.965955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.965959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.966623 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.967089 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:39.967094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:39.967102 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:39.967296 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:39.967967 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:39.967972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:39.968831 (+ 859us) client_negotiation.cc:769] Sending connection context
0929 11:33:39.968995 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:33:39.969328 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11260,"thread_start_us":11134,"threads_started":1}
I20260929 11:33:39.969779 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:39.941619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:50647 (local address 127.6.9.254:46741)
0929 11:33:39.951814 (+ 10195us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:39.951820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:39.953148 (+ 1328us) server_negotiation.cc:407] Connection header received
0929 11:33:39.953219 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:39.953224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:39.953306 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:39.953510 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:39.954751 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.955750 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.957516 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:39.958150 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:39.958268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:39.962037 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:39.962044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:39.962048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:39.962090 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:39.965721 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.966759 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.966764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.966768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.966948 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:39.967426 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:39.967431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:39.967434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:39.967770 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:39.967906 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:39.969152 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0929 11:33:39.969383 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10055,"thread_start_us":10204,"threads_started":1}
I20260929 11:33:39.970196 13418 heartbeater.cc:347] Connected to a master server at 127.6.9.253:33433
I20260929 11:33:39.970554 13418 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.970911 13416 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:39.971071 13416 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.971568 13417 heartbeater.cc:347] Connected to a master server at 127.6.9.252:37129
I20260929 11:33:39.971627 13416 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
I20260929 11:33:39.971669 13418 heartbeater.cc:511] Master 127.6.9.253:33433 requested a full tablet report, sending...
I20260929 11:33:39.971740 13417 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:39.972291 13417 heartbeater.cc:511] Master 127.6.9.252:37129 requested a full tablet report, sending...
I20260929 11:33:39.973562 12302 ts_manager.cc:194] Re-registered known tserver with Master: b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:39.973640 12899 ts_manager.cc:194] Registered new tserver with Master: b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:39.974567 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:50647
I20260929 11:33:39.975490 13088 ts_manager.cc:194] Registered new tserver with Master: b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:40.057979 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12630
I20260929 11:33:40.072000 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:43307
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:46741
--webserver_port=45393
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:33:40.358721 13265 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
W20260929 11:33:40.413226 13429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:40.413506 13429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:40.413646 13429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:40.420799 13429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:40.420921 13429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:40.421105 13429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:40.431277 13429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:43307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=45393
--tserver_master_addrs=127.6.9.254:46741,127.6.9.253:33433,127.6.9.252:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:40.432718 13429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:40.434130 13429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:40.447656 13434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:40.447575 13435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:40.447600 13437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:40.448498 13429 server_base.cc:1034] running on GCE node
I20260929 11:33:40.449187 13429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:40.450210 13429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:40.451414 13429 hybrid_clock.cc:648] HybridClock initialized: now 1790681620451347 us; error 60 us; skew 500 ppm
I20260929 11:33:40.451738 13429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:40 dist-test-slave-c0l9 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681620, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:40.454875 13429 init.cc:377] Logged in from keytab as oryx/127.6.9.195@KRBTEST.COM (short username oryx)
I20260929 11:33:40.456331 13429 webserver.cc:460] Webserver started at http://127.6.9.195:45393/ using document root <none> and password file <none>
I20260929 11:33:40.457070 13429 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:40.457221 13429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:40.462889 13429 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 11:33:40.465830 13444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:40.467303 13429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:33:40.467485 13429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "36ef156af6f84be5957cefb702865298"
format_stamp: "Formatted at 2026-09-29 11:33:37 on dist-test-slave-c0l9"
server_key: "6589924d49bd401097d379667c253459"
server_key_iv: "d2d1e5730e9206f35ba7efb97c06a81d"
server_key_version: "encryptionkey@0"
I20260929 11:33:40.468479 13429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:40.484714 13429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:40.489040 13429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:40.489310 13429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:40.490332 13429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:40.491750 13429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:40.491846 13429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:40.491956 13429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:40.492035 13429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:40.522183 13429 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:43307
I20260929 11:33:40.522248 13556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:43307 every 8 connection(s)
I20260929 11:33:40.523591 13429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:40.531690 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13429
Sep 29 11:33:40 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681620, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
Sep 29 11:33:40 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681620, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.252@KRBTEST.COM
Sep 29 11:33:40 dist-test-slave-c0l9 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681620, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.253@KRBTEST.COM
I20260929 11:33:40.570909 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.526540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:37961 (local address 127.6.9.252:37129)
0929 11:33:40.545553 (+ 19013us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.545557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.545575 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:40.546473 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.546481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.546586 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.546697 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.549533 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.550752 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.553681 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.554308 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.554432 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.562456 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.562465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.562470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.562516 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.566463 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.567731 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.567738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.567743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.567922 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.568558 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.568563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.568566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.568966 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.569108 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.570243 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.570492 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18908,"thread_start_us":18808,"threads_started":1}
I20260929 11:33:40.573485 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.527293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.6.9.195:37961)
0929 11:33:40.539330 (+ 12037us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.539400 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.545859 (+ 6459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.547220 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.547233 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.548231 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:40.549316 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.549335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.550950 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.550954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.553491 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.553508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.554765 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.554772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.555149 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.556026 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:40.556068 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:40.562126 (+ 6058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:40.566676 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.566689 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.566714 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.567416 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.568092 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.568098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.568101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.568423 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.569208 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:40.569219 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:40.569767 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.571307 (+ 1540us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.571931 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11617,"mutex_wait_us":328,"spinlock_wait_cycles":17792,"thread_start_us":11365,"threads_started":1}
I20260929 11:33:40.575309 13558 heartbeater.cc:347] Connected to a master server at 127.6.9.252:37129
I20260929 11:33:40.575646 13558 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:40.576414 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.546026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.6.9.195:38811)
0929 11:33:40.546725 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.546744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.546862 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.549467 (+ 2605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.549472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.550072 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:40.550525 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.550549 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.552093 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.552099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.554083 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.554101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.555379 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.555386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.555619 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.556239 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:40.556269 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:40.565528 (+ 9259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:40.570332 (+ 4804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.570338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.570342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.572810 (+ 2468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.573250 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.573256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.573259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.573466 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.575344 (+ 1878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:40.575349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:40.575502 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.575682 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.575989 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":35,"thread_start_us":512,"threads_started":1}
I20260929 11:33:40.576468 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.546146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:38811 (local address 127.6.9.253:33433)
0929 11:33:40.547291 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.547298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.547318 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:33:40.549033 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.549041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.549152 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.549276 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.550734 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.551908 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.554268 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.554979 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.555098 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.566553 (+ 11455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.566561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.566566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.566622 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.570154 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.572933 (+ 2779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.572939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.572942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.573101 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.574746 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.574752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.574756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.575124 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.575257 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.575834 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.576083 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":910,"threads_started":1}
I20260929 11:33:40.576918 13558 heartbeater.cc:511] Master 127.6.9.252:37129 requested a full tablet report, sending...
I20260929 11:33:40.578214 13559 heartbeater.cc:347] Connected to a master server at 127.6.9.253:33433
I20260929 11:33:40.578351 13559 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:40.578786 13559 heartbeater.cc:511] Master 127.6.9.253:33433 requested a full tablet report, sending...
I20260929 11:33:40.578836 13088 ts_manager.cc:194] Registered new tserver with Master: 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:40.580057 12899 ts_manager.cc:194] Registered new tserver with Master: 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:40.583988 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.527293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:48191)
0929 11:33:40.545783 (+ 18490us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.545808 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.545967 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.547511 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.547517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.548247 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:40.549305 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.549335 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.550913 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.550919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.553491 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.553508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.554765 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.554772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.555144 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.556290 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:40.556311 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:40.570913 (+ 14602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:40.580640 (+ 9727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.580645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.580650 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.581210 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.581905 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:40.581910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:40.581913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:40.582095 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:40.582904 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:40.582910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:40.583078 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.583246 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.583552 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18310,"mutex_wait_us":311,"spinlock_wait_cycles":41856,"thread_start_us":17974,"threads_started":1}
I20260929 11:33:40.584968 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.526546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:48191 (local address 127.6.9.254:46741)
0929 11:33:40.545205 (+ 18659us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.545213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.547216 (+ 2003us) server_negotiation.cc:407] Connection header received
0929 11:33:40.547285 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.547290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.547366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.547484 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.549538 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.550734 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.553681 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.554399 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.554500 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.571163 (+ 16663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.571170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.571263 (+ 93us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.571306 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.580464 (+ 9158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.581574 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.581580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.581583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.581766 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.582255 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.582260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.582263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.582674 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.582833 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.584346 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.584578 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18483,"thread_start_us":18348,"threads_started":1}
I20260929 11:33:40.585819 13557 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46741
I20260929 11:33:40.585976 13557 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:40.586383 13557 heartbeater.cc:511] Master 127.6.9.254:46741 requested a full tablet report, sending...
I20260929 11:33:40.587276 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.560035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.6.9.252:37129)
0929 11:33:40.560773 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.560779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.560805 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:33:40.560867 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.560873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.560952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.561072 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.564094 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.565517 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.567449 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.568120 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.568242 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.579930 (+ 11688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.579937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.579942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.579981 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.583327 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.584295 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.584301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.584305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.584465 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.584962 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.584967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.584970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.585804 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.586034 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.586403 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.586629 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":480,"threads_started":1}
I20260929 11:33:40.587433 12302 ts_manager.cc:194] Re-registered known tserver with Master: 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:40.588518 12302 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:48191
I20260929 11:33:40.588969 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.559974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.6.9.254:46741)
0929 11:33:40.568795 (+ 8821us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.568802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.568821 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:33:40.568882 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.568888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.568970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.570841 (+ 1871us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.571667 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.573150 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.575252 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.575868 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.576574 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.579655 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.579662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.579667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.579706 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.583664 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.584728 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.584737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.584741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.584866 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.585273 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.585277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.585280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.585622 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.588270 (+ 2648us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.588389 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.588613 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8670,"thread_start_us":8527,"threads_started":1}
I20260929 11:33:40.596849 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.559631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.6.9.253:33433)
0929 11:33:40.561663 (+ 2032us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.561669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.581228 (+ 19559us) server_negotiation.cc:407] Connection header received
0929 11:33:40.581489 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.581494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.581572 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.581661 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:40.583631 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.584877 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.587145 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.587756 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.587856 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.591727 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:40.591732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:40.591736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:40.591770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:40.594219 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.594889 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.594895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.594898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.595043 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:40.595391 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:40.595395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:40.595397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:40.595680 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:40.595917 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.596250 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.596471 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":1760,"threads_started":1}
I20260929 11:33:40.593146 12302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:40.600311 12302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:40.638587 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.619295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:43307 (local address 127.0.0.1:49128)
0929 11:33:40.621000 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.621023 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.621165 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.630212 (+ 9047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.630219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.630259 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:40.630803 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.630820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.632618 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.632624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.634306 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.634322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.637681 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.637687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.637892 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.637927 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.638006 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.638155 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":754,"threads_started":1}
I20260929 11:33:40.639835 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.619896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.6.9.195:43307)
0929 11:33:40.629331 (+ 9435us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.629342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.629388 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:33:40.629528 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.629537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.629876 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.630170 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:40.631076 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.632467 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.634486 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.637443 (+ 2957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.638995 (+ 1552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.639232 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.639316 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.639489 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9193,"thread_start_us":8999,"threads_started":1}
I20260929 11:33:40.642412 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.619600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42045 (local address 127.0.0.1:50254)
0929 11:33:40.634643 (+ 15043us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.634660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.634772 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.635687 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.635693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.635727 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:40.636089 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.636100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.637788 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.637793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.639663 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.639677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.641574 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.641581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.641756 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.641790 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.641909 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.642049 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14936,"thread_start_us":15205,"threads_started":1}
I20260929 11:33:40.643226 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.619828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.6.9.193:42045)
0929 11:33:40.630920 (+ 11092us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.630927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.634782 (+ 3855us) server_negotiation.cc:407] Connection header received
0929 11:33:40.634917 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.634927 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.635213 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.635435 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:40.636296 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.637633 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.639829 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.641352 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.641483 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.641825 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.642740 (+ 915us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.642940 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10926,"thread_start_us":10797,"threads_started":1}
I20260929 11:33:40.645382 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.635159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:33983 (local address 127.0.0.1:37764)
0929 11:33:40.635757 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.635771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.635870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.637736 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.637741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.637772 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:40.638462 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.638476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.640194 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.640200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.642810 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.642823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.644533 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.644540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.644763 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.644799 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.644887 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.645031 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260929 11:33:40.645416 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.635404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.6.9.194:33983)
0929 11:33:40.637081 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.637090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.637120 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:33:40.637245 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.637253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.637489 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.637883 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:40.638655 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.640063 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.642974 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.644304 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.644474 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.644709 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.644958 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.645131 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1515,"thread_start_us":1415,"threads_started":1}
I20260929 11:33:40.644058 13492 tablet_service.cc:1467] Processing CreateTablet for tablet 281916f5d4154d37a5379550e914e7f2 (DEFAULT_TABLE table=test-table [id=e3bc2cb92e334e4f8009a1e56525ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:40.646737 13492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281916f5d4154d37a5379550e914e7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:40.647305 13200 tablet_service.cc:1467] Processing CreateTablet for tablet 281916f5d4154d37a5379550e914e7f2 (DEFAULT_TABLE table=test-table [id=e3bc2cb92e334e4f8009a1e56525ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:40.648815 13200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281916f5d4154d37a5379550e914e7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:40.648792 13351 tablet_service.cc:1467] Processing CreateTablet for tablet 281916f5d4154d37a5379550e914e7f2 (DEFAULT_TABLE table=test-table [id=e3bc2cb92e334e4f8009a1e56525ace6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:40.649868 13351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 281916f5d4154d37a5379550e914e7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:40.661397 13591 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Bootstrap starting.
I20260929 11:33:40.664587 13592 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Bootstrap starting.
I20260929 11:33:40.665139 13591 tablet_bootstrap.cc:654] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:40.666496 13591 log.cc:824] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:40.667835 13593 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Bootstrap starting.
I20260929 11:33:40.668102 13592 tablet_bootstrap.cc:654] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:40.669339 13592 log.cc:824] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:40.670305 13591 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: No bootstrap required, opened a new log
I20260929 11:33:40.670614 13591 ts_tablet_manager.cc:1397] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 11:33:40.671123 13593 tablet_bootstrap.cc:654] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:40.672294 13593 log.cc:824] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:40.673985 13592 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: No bootstrap required, opened a new log
I20260929 11:33:40.674276 13592 ts_tablet_manager.cc:1397] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:33:40.678225 13591 raft_consensus.cc:348] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.678586 13591 raft_consensus.cc:374] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:40.678679 13591 raft_consensus.cc:729] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98d47edc02a42d59ef41d5553f37fb1, State: Initialized, Role: FOLLOWER
I20260929 11:33:40.679224 13591 consensus_queue.cc:260] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.679641 13593 tablet_bootstrap.cc:492] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: No bootstrap required, opened a new log
I20260929 11:33:40.679915 13593 ts_tablet_manager.cc:1397] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260929 11:33:40.680315 13416 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
I20260929 11:33:40.680943 13591 ts_tablet_manager.cc:1428] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260929 11:33:40.686988 13592 raft_consensus.cc:348] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.687470 13592 raft_consensus.cc:374] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:40.687575 13592 raft_consensus.cc:729] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef156af6f84be5957cefb702865298, State: Initialized, Role: FOLLOWER
I20260929 11:33:40.688190 13592 consensus_queue.cc:260] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
W20260929 11:33:40.689177 13419 tablet.cc:2369] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:40.689364 13557 heartbeater.cc:503] Master 127.6.9.254:46741 was elected leader, sending a full tablet report...
I20260929 11:33:40.689385 13593 raft_consensus.cc:348] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.689807 13593 raft_consensus.cc:374] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:40.689901 13593 raft_consensus.cc:729] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a21eea0381e4479943d18ef9d9dc7c8, State: Initialized, Role: FOLLOWER
I20260929 11:33:40.690040 13592 ts_tablet_manager.cc:1428] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260929 11:33:40.690618 13593 consensus_queue.cc:260] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.695142 13593 ts_tablet_manager.cc:1428] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260929 11:33:40.776510 13560 tablet.cc:2369] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:33:40.800844 13268 tablet.cc:2369] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:40.954144 13598 raft_consensus.cc:482] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:40.954469 13598 raft_consensus.cc:504] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.956143 13598 leader_election.cc:288] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045), b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:40.961601 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.956633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:51579 (local address 127.6.9.193:42045)
0929 11:33:40.956864 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.956870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.956892 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:33:40.956969 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.956974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.957048 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.957150 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:40.957502 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.958204 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.959977 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.960770 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.960882 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.960981 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.961248 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.961371 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:33:40.961622 13564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.956451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42045 (local address 127.6.9.195:51579)
0929 11:33:40.956643 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.956684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.956804 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.957191 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.957196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.957221 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:40.957370 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.957377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.958326 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.958331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.959829 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.959842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.960903 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.960909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.961095 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.961130 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.961210 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.961352 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":7}
I20260929 11:33:40.961681 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.956814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:33983 (local address 127.6.9.195:49189)
0929 11:33:40.957007 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:40.957023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:40.957112 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:40.957434 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:40.957438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:40.957461 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:40.957626 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.957634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.958584 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.958588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.959886 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:40.959898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.960986 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.960992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:40.961169 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.961191 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 11:33:40.961266 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:33:40.961405 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":24}
I20260929 11:33:40.961661 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.956966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:49189 (local address 127.6.9.194:33983)
0929 11:33:40.957188 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.957194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.957212 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:40.957269 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.957273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.957325 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.957398 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:40.957758 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.958440 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.960021 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.960825 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.960935 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.961038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.961305 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.961426 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:33:40.962698 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281916f5d4154d37a5379550e914e7f2" candidate_uuid: "36ef156af6f84be5957cefb702865298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98d47edc02a42d59ef41d5553f37fb1" is_pre_election: true
I20260929 11:33:40.962699 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281916f5d4154d37a5379550e914e7f2" candidate_uuid: "36ef156af6f84be5957cefb702865298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" is_pre_election: true
I20260929 11:33:40.963255 13371 raft_consensus.cc:2454] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef156af6f84be5957cefb702865298 in term 0.
I20260929 11:33:40.963255 13220 raft_consensus.cc:2454] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef156af6f84be5957cefb702865298 in term 0.
I20260929 11:33:40.963923 13445 leader_election.cc:302] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef156af6f84be5957cefb702865298, b98d47edc02a42d59ef41d5553f37fb1; no voters:
I20260929 11:33:40.964304 13598 raft_consensus.cc:2781] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:40.964437 13598 raft_consensus.cc:482] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:40.964541 13598 raft_consensus.cc:3037] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:40.966706 13598 raft_consensus.cc:504] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.967235 13598 leader_election.cc:288] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [CANDIDATE]: Term 1 election: Requested vote from peers 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045), b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983)
I20260929 11:33:40.967706 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281916f5d4154d37a5379550e914e7f2" candidate_uuid: "36ef156af6f84be5957cefb702865298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
I20260929 11:33:40.967742 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "281916f5d4154d37a5379550e914e7f2" candidate_uuid: "36ef156af6f84be5957cefb702865298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98d47edc02a42d59ef41d5553f37fb1"
I20260929 11:33:40.967923 13371 raft_consensus.cc:3037] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:40.967924 13220 raft_consensus.cc:3037] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:40.971082 13220 raft_consensus.cc:2454] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef156af6f84be5957cefb702865298 in term 1.
I20260929 11:33:40.971089 13371 raft_consensus.cc:2454] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef156af6f84be5957cefb702865298 in term 1.
I20260929 11:33:40.971520 13445 leader_election.cc:302] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef156af6f84be5957cefb702865298, b98d47edc02a42d59ef41d5553f37fb1; no voters:
I20260929 11:33:40.971788 13598 raft_consensus.cc:2781] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:40.972189 13598 raft_consensus.cc:686] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 1 LEADER]: Becoming Leader. State: Replica: 36ef156af6f84be5957cefb702865298, State: Running, Role: LEADER
I20260929 11:33:40.972751 13598 consensus_queue.cc:237] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } }
I20260929 11:33:40.977809 12302 catalog_manager.cc:5495] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ef156af6f84be5957cefb702865298 (127.6.9.195). New cstate: current_term: 1 leader_uuid: "36ef156af6f84be5957cefb702865298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:40.999529 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:40.994476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.6.9.195:43307)
0929 11:33:40.994692 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:40.994698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:40.994717 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:33:40.994806 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:40.994810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:40.994885 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:40.995012 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:40.995376 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.996056 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.997400 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:40.997822 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:40.997934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:40.998408 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:40.998932 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:40.999076 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:40.999179 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 11:33:40.999281 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:33:41.007947 13371 raft_consensus.cc:1261] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Refusing update from remote peer 36ef156af6f84be5957cefb702865298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:41.007947 13220 raft_consensus.cc:1261] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Refusing update from remote peer 36ef156af6f84be5957cefb702865298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:41.009003 13600 consensus_queue.cc:1035] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:41.009346 13598 consensus_queue.cc:1035] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:41.033674 13604 mvcc.cc:205] Tried to move back new op lower bound from 7334631919637078016 to 7334631919508205568. Current Snapshot: MvccSnapshot[applied={T|T < 7334631919637078016}]
I20260929 11:33:41.038512 12302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:41.038831 12302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:41.046454 12302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 715519d5829c48c48a4bcdfe3a49bfbf: Sending DeleteTablet for 3 replicas of tablet 281916f5d4154d37a5379550e914e7f2
I20260929 11:33:41.048223 13200 tablet_service.cc:1514] Processing DeleteTablet for tablet 281916f5d4154d37a5379550e914e7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:41 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:50254
I20260929 11:33:41.048272 13351 tablet_service.cc:1514] Processing DeleteTablet for tablet 281916f5d4154d37a5379550e914e7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:41 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:37764
I20260929 11:33:41.049029 13492 tablet_service.cc:1514] Processing DeleteTablet for tablet 281916f5d4154d37a5379550e914e7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:41 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:49128
I20260929 11:33:41.059646 13614 tablet_replica.cc:330] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: stopping tablet replica
I20260929 11:33:41.061717 13614 raft_consensus.cc:2229] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:41.062465 13614 raft_consensus.cc:2258] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:41.063894 13615 tablet_replica.cc:330] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: stopping tablet replica
I20260929 11:33:41.064419 13615 raft_consensus.cc:2229] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:41.065022 13615 raft_consensus.cc:2258] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:41.065043 13616 tablet_replica.cc:330] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: stopping tablet replica
I20260929 11:33:41.065522 13616 raft_consensus.cc:2229] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:41.066272 13616 raft_consensus.cc:2258] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:41.068816 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12270
I20260929 11:33:41.070093 12332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:41.070078 13615 ts_tablet_manager.cc:1905] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:41.074455 13616 ts_tablet_manager.cc:1905] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:41.067616 13614 ts_tablet_manager.cc:1905] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:41.084383 13614 ts_tablet_manager.cc:1918] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:41.084612 13614 log.cc:1196] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/281916f5d4154d37a5379550e914e7f2
I20260929 11:33:41.084808 13616 ts_tablet_manager.cc:1918] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:41.084985 13616 log.cc:1196] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/281916f5d4154d37a5379550e914e7f2
I20260929 11:33:41.085248 13614 ts_tablet_manager.cc:1939] T 281916f5d4154d37a5379550e914e7f2 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting consensus metadata
I20260929 11:33:41.085528 13616 ts_tablet_manager.cc:1939] T 281916f5d4154d37a5379550e914e7f2 P 36ef156af6f84be5957cefb702865298: Deleting consensus metadata
I20260929 11:33:41.089772 13615 ts_tablet_manager.cc:1918] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:41.089977 13615 log.cc:1196] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/281916f5d4154d37a5379550e914e7f2
I20260929 11:33:41.090466 13615 ts_tablet_manager.cc:1939] T 281916f5d4154d37a5379550e914e7f2 P b98d47edc02a42d59ef41d5553f37fb1: Deleting consensus metadata
I20260929 11:33:41.101356 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:41.050074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.6.9.253:33433)
0929 11:33:41.050262 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:41.050269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:41.067307 (+ 17038us) server_negotiation.cc:407] Connection header received
0929 11:33:41.067658 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:41.067665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:41.067757 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:41.067838 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:41.070238 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.071246 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.077424 (+ 6178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.078088 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.078219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:41.095719 (+ 17500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:41.095724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:41.095728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:41.095771 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:41.098531 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.099297 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.099302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.099304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.099438 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.099860 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.099864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.099866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.100170 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:41.100391 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:41.100778 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 11:33:41.101014 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:33:41.113998 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:41.067616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.6.9.252:37129)
0929 11:33:41.067827 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:41.067834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:41.101862 (+ 34028us) server_negotiation.cc:407] Connection header received
0929 11:33:41.102112 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:41.102118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:41.102201 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:41.102270 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:41.103756 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.104448 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.105708 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.106110 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.106219 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:41.108814 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:41.108820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:41.108824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:41.108866 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:41.111411 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.112031 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.112035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.112037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.112167 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.112517 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.112520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.112522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.112829 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:41.113039 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:41.113355 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:33:41.113573 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:33:41.231655 12289 catalog_manager.cc:4841] TS 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307): tablet 281916f5d4154d37a5379550e914e7f2 (table test-table [id=e3bc2cb92e334e4f8009a1e56525ace6]) successfully deleted
W20260929 11:33:41.232074 12289 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.9.195:43307, user_credentials={real_user=slave}} blocked reactor thread for 144738us
I20260929 11:33:41.232654 12286 catalog_manager.cc:4841] TS 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045): tablet 281916f5d4154d37a5379550e914e7f2 (table test-table [id=e3bc2cb92e334e4f8009a1e56525ace6]) successfully deleted
W20260929 11:33:41.232857 12286 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.9.193:42045, user_credentials={real_user=slave}} blocked reactor thread for 145547us
I20260929 11:33:41.233089 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12270
I20260929 11:33:41.282451 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:41.265425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.6.9.252:37129)
0929 11:33:41.265639 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:41.265646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:41.266038 (+ 392us) server_negotiation.cc:407] Connection header received
0929 11:33:41.266312 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:41.266318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:41.266400 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:41.266471 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:41.268834 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.269934 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.271994 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.272656 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.272968 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:41.276519 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:41.276524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:41.276528 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:41.276564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:41.279505 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.280218 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.280224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.280228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.280420 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.280871 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.280876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.280878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.281223 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:41.281449 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:41.281872 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 11:33:41.282101 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:33:41.288604 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:41.265892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.6.9.253:33433)
0929 11:33:41.266071 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:41.266078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:41.274701 (+ 8623us) server_negotiation.cc:407] Connection header received
0929 11:33:41.274888 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:41.274893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:41.274967 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:41.275049 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:41.276505 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.277642 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.279414 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:41.280085 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:41.280203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:41.283285 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:41.283290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:41.283293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:41.283326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:41.286020 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.286676 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.286682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.286686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.286827 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:41.287187 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:41.287192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:41.287196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:41.287477 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:41.287711 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:41.288035 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:33:41.288264 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:33:42.032380 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.031399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:40259)
0929 11:33:42.032018 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.032164 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":401,"threads_started":1}
W20260929 11:33:42.032861 13557 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46741 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.034591 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.033958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:44743)
0929 11:33:42.034343 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.034458 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":190,"threads_started":1}
W20260929 11:33:42.034920 13416 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46741 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.038540 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.037821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:35401)
0929 11:33:42.038220 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.038351 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":203,"threads_started":1}
W20260929 11:33:42.038938 13265 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46741 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.546680 13629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 715519d5829c48c48a4bcdfe3a49bfbf)
I20260929 11:33:42.546864 13630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 715519d5829c48c48a4bcdfe3a49bfbf)
I20260929 11:33:42.547011 13629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:42.547118 13630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:42.548465 13630 leader_election.cc:288] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253:33433)
I20260929 11:33:42.548477 13629 leader_election.cc:288] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 91dee2623f154dd1b596f7625f8641ee (127.6.9.252:37129)
I20260929 11:33:42.549885 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.548649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60286)
0929 11:33:42.549403 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.549596 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":470,"threads_started":1}
W20260929 11:33:42.550207 12886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.556353 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.548805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60278)
0929 11:33:42.549706 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.556026 (+ 6320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":652,"threads_started":1}
W20260929 11:33:42.559453 13075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.565302 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.564624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60294)
0929 11:33:42.564875 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.565058 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:33:42.565953 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.565320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60306)
0929 11:33:42.565515 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.565726 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:33:42.566200 13075 leader_election.cc:334] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
W20260929 11:33:42.566167 12886 leader_election.cc:334] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.566627 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.548881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:37129 (local address 127.0.0.1:42924)
0929 11:33:42.556388 (+ 7507us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.556414 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:42.556587 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:42.559934 (+ 3347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:42.559943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:42.559970 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:42.560331 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:42.560345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.562182 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.562188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:42.564403 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:42.564416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.565919 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.565925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:42.566105 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:42.566142 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:33:42.566221 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 11:33:42.566368 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7394,"thread_start_us":9675,"threads_started":1}
I20260929 11:33:42.566912 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.549960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:33433 (local address 127.0.0.1:41108)
0929 11:33:42.557310 (+ 7350us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.557331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:42.557485 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:42.558921 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:42.558929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:42.558966 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:42.559278 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:42.559294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.561312 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.561318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:42.563612 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:42.563630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.565786 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.565795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:42.565979 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:42.566016 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:33:42.566478 (+ 462us) client_negotiation.cc:240] Negotiation successful
0929 11:33:42.566625 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7159,"thread_start_us":8647,"threads_started":1}
I20260929 11:33:42.566962 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.549928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.6.9.253:33433)
0929 11:33:42.558595 (+ 8667us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:42.558602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:42.558627 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:33:42.558699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:42.558704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:42.558777 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:42.558886 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:42.559711 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.561100 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.564377 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.565613 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.565737 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:42.565835 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:42.566508 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 11:33:42.566645 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8485,"thread_start_us":8287,"threads_started":1}
I20260929 11:33:42.567093 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.558916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.6.9.252:37129)
0929 11:33:42.559406 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:42.559412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:42.559431 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:33:42.559496 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:42.559500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:42.559580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:42.559692 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:42.560558 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.561926 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.564534 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:42.565696 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:42.566453 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:42.566574 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:42.566617 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 11:33:42.566740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":613,"server-negotiator.queue_time_us":379,"thread_start_us":274,"threads_started":1}
I20260929 11:33:42.567601 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91dee2623f154dd1b596f7625f8641ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee440cc6944cbe961cd9955592ed22" is_pre_election: true
I20260929 11:33:42.567602 13098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4cee440cc6944cbe961cd9955592ed22" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91dee2623f154dd1b596f7625f8641ee" is_pre_election: true
I20260929 11:33:42.567891 13098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cee440cc6944cbe961cd9955592ed22 in term 1.
I20260929 11:33:42.567885 12909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dee2623f154dd1b596f7625f8641ee in term 1.
I20260929 11:33:42.568312 13075 leader_election.cc:302] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cee440cc6944cbe961cd9955592ed22, 91dee2623f154dd1b596f7625f8641ee; no voters: 715519d5829c48c48a4bcdfe3a49bfbf
I20260929 11:33:42.568310 12883 leader_election.cc:302] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cee440cc6944cbe961cd9955592ed22, 91dee2623f154dd1b596f7625f8641ee; no voters: 715519d5829c48c48a4bcdfe3a49bfbf
I20260929 11:33:42.568612 13630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 11:33:42.568650 13629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 11:33:42.568756 13630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 715519d5829c48c48a4bcdfe3a49bfbf)
I20260929 11:33:42.568800 13629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 715519d5829c48c48a4bcdfe3a49bfbf)
I20260929 11:33:42.568846 13630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:33:42.568883 13629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:33:42.570901 13629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:42.570919 13630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:42.571467 13630 leader_election.cc:288] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 election: Requested vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253:33433)
I20260929 11:33:42.571493 13629 leader_election.cc:288] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 election: Requested vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 91dee2623f154dd1b596f7625f8641ee (127.6.9.252:37129)
I20260929 11:33:42.572108 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.571622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60310)
0929 11:33:42.571774 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.571866 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 11:33:42.572120 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.571668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60324)
0929 11:33:42.571845 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.571933 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 11:33:42.572216 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91dee2623f154dd1b596f7625f8641ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee440cc6944cbe961cd9955592ed22"
I20260929 11:33:42.572436 13098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4cee440cc6944cbe961cd9955592ed22" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91dee2623f154dd1b596f7625f8641ee"
I20260929 11:33:42.572683 12909 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 91dee2623f154dd1b596f7625f8641ee in current term 2: Already voted for candidate 4cee440cc6944cbe961cd9955592ed22 in this term.
I20260929 11:33:42.572803 13098 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cee440cc6944cbe961cd9955592ed22 in current term 2: Already voted for candidate 91dee2623f154dd1b596f7625f8641ee in this term.
I20260929 11:33:42.573333 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.572850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60326)
0929 11:33:42.573050 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.573204 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 11:33:42.573567 13075 leader_election.cc:334] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.574065 13075 leader_election.cc:302] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91dee2623f154dd1b596f7625f8641ee; no voters: 4cee440cc6944cbe961cd9955592ed22, 715519d5829c48c48a4bcdfe3a49bfbf
I20260929 11:33:42.574151 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:42.573562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60340)
0929 11:33:42.573900 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:42.574001 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
W20260929 11:33:42.574333 12886 leader_election.cc:334] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:42.574400 13630 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 11:33:42.574497 12886 leader_election.cc:302] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cee440cc6944cbe961cd9955592ed22; no voters: 715519d5829c48c48a4bcdfe3a49bfbf, 91dee2623f154dd1b596f7625f8641ee
I20260929 11:33:42.574813 13629 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 11:33:43.034715 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:43.033530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:59227)
0929 11:33:43.034373 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:43.034483 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":343,"threads_started":1}
I20260929 11:33:43.036201 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:43.035510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:34277)
0929 11:33:43.035962 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:43.036038 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":263,"threads_started":1}
I20260929 11:33:43.040329 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:43.039581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:36243)
0929 11:33:43.040029 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:43.040117 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260929 11:33:44.036594 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.035705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:41301)
0929 11:33:44.036230 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.036364 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260929 11:33:44.038056 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.037357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:60041)
0929 11:33:44.037840 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.037916 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":294,"threads_started":1}
I20260929 11:33:44.041889 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.041163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:39105)
0929 11:33:44.041602 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.041701 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
I20260929 11:33:44.739993 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:44.740202 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:44.740887 13645 leader_election.cc:288] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253:33433)
I20260929 11:33:44.742084 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.741131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60612)
0929 11:33:44.741631 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.741824 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":233,"threads_started":1}
I20260929 11:33:44.742111 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91dee2623f154dd1b596f7625f8641ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee440cc6944cbe961cd9955592ed22" is_pre_election: true
I20260929 11:33:44.742376 12909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dee2623f154dd1b596f7625f8641ee in term 2.
I20260929 11:33:44.742894 13075 leader_election.cc:302] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cee440cc6944cbe961cd9955592ed22, 91dee2623f154dd1b596f7625f8641ee; no voters:
I20260929 11:33:44.743155 13645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 11:33:44.743264 13645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:44.743360 13645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:33:44.743713 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.743301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60624)
0929 11:33:44.743489 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.743570 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 11:33:44.743880 13075 leader_election.cc:334] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:44.745548 13645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:44.746135 13645 leader_election.cc:288] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 election: Requested vote from peers 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741), 4cee440cc6944cbe961cd9955592ed22 (127.6.9.253:33433)
I20260929 11:33:44.746605 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.746253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60636)
0929 11:33:44.746373 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.746482 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 11:33:44.746842 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91dee2623f154dd1b596f7625f8641ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4cee440cc6944cbe961cd9955592ed22"
I20260929 11:33:44.747062 12909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:33:44.747617 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.747264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60650)
0929 11:33:44.747396 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.747477 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 11:33:44.747786 13075 leader_election.cc:334] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
I20260929 11:33:44.749401 12909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91dee2623f154dd1b596f7625f8641ee in term 3.
I20260929 11:33:44.749845 13075 leader_election.cc:302] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cee440cc6944cbe961cd9955592ed22, 91dee2623f154dd1b596f7625f8641ee; no voters: 715519d5829c48c48a4bcdfe3a49bfbf
I20260929 11:33:44.750105 13645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:33:44.750479 13645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [term 3 LEADER]: Becoming Leader. State: Replica: 91dee2623f154dd1b596f7625f8641ee, State: Running, Role: LEADER
I20260929 11:33:44.750891 13645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } }
I20260929 11:33:44.753233 13648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91dee2623f154dd1b596f7625f8641ee. Latest consensus state: current_term: 3 leader_uuid: "91dee2623f154dd1b596f7625f8641ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:44.753515 13648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:44.754104 13650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:44.757635 13650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:44.758417 13650 catalog_manager.cc:1259] Loaded cluster ID: e87b41d2600742d9a2565ab6f49a0807
I20260929 11:33:44.758494 13650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:44.759608 13650 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:44.760375 13650 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Loaded TSK: 0
I20260929 11:33:44.761267 13650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:33:44.810103 13087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:44.817329 13087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:44.821849 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.821402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60668)
0929 11:33:44.821596 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.821694 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:33:44.822314 12909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [term 3 FOLLOWER]: Refusing update from remote peer 91dee2623f154dd1b596f7625f8641ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 11:33:44.822913 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.822536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60684)
0929 11:33:44.822678 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.822765 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 11:33:44.823055 13075 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee -> Peer 715519d5829c48c48a4bcdfe3a49bfbf (127.6.9.254:46741): Couldn't send request to peer 715519d5829c48c48a4bcdfe3a49bfbf. Status: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:33:44.823078 13645 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 11:33:44.826434 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91dee2623f154dd1b596f7625f8641ee. Latest consensus state: current_term: 3 leader_uuid: "91dee2623f154dd1b596f7625f8641ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:44.826735 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:44.827617 13645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "91dee2623f154dd1b596f7625f8641ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:44.827901 13645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:44.828392 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "91dee2623f154dd1b596f7625f8641ee" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "715519d5829c48c48a4bcdfe3a49bfbf" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46741 } } peers { permanent_uuid: "4cee440cc6944cbe961cd9955592ed22" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 33433 } attrs { promote: false } } peers { permanent_uuid: "91dee2623f154dd1b596f7625f8641ee" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 37129 } attrs { promote: false } } }
I20260929 11:33:44.828706 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cee440cc6944cbe961cd9955592ed22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:33:44.848063 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.836747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42045 (local address 127.0.0.1:58886)
0929 11:33:44.836896 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.836914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:44.837034 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:44.841608 (+ 4574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:44.841613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:44.841639 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:44.842012 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.842022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.843595 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.843600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.845642 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.845658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.847228 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.847235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.847405 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.847462 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 11:33:44.847531 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 11:33:44.847674 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":73}
I20260929 11:33:44.849205 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.836869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.6.9.193:42045)
0929 11:33:44.841293 (+ 4424us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:44.841299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:44.841317 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:44.841385 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:44.841391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:44.841462 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:44.842128 (+ 666us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:44.842311 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.843468 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.845839 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.847029 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.848624 (+ 1595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.848764 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:44.848817 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 11:33:44.848946 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4295,"thread_start_us":4215,"threads_started":1}
I20260929 11:33:44.849326 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.837099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:33983 (local address 127.0.0.1:44584)
0929 11:33:44.837663 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.837680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:44.837791 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:44.841456 (+ 3665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:44.841461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:44.841489 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:44.841927 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.841941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.843541 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.843546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.846362 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.846374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.847984 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.847990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.848373 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.848406 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:33:44.848460 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 11:33:44.848595 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":64,"spinlock_wait_cycles":1280,"thread_start_us":509,"threads_started":1}
I20260929 11:33:44.850198 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.837490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.6.9.194:33983)
0929 11:33:44.841099 (+ 3609us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:44.841106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:44.841127 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:33:44.841204 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:44.841209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:44.841287 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:44.842129 (+ 842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:44.842327 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.843373 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.846541 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.847786 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.849676 (+ 1890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.849759 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:44.849804 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 11:33:44.849927 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3485,"thread_start_us":3381,"threads_started":1}
I20260929 11:33:44.850262 13200 tablet_service.cc:1467] Processing CreateTablet for tablet b091332db96b40f7895fd46f1b9408c0 (DEFAULT_TABLE table=test-table [id=e62516a205e94ee8ac56932a88eae12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:44.850770 13200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091332db96b40f7895fd46f1b9408c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:44.851204 13351 tablet_service.cc:1467] Processing CreateTablet for tablet b091332db96b40f7895fd46f1b9408c0 (DEFAULT_TABLE table=test-table [id=e62516a205e94ee8ac56932a88eae12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:44.851622 13351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091332db96b40f7895fd46f1b9408c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:44.856597 13663 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Bootstrap starting.
I20260929 11:33:44.857758 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.837603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:43307 (local address 127.0.0.1:54200)
0929 11:33:44.848326 (+ 10723us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.848345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:44.848442 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:44.851704 (+ 3262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:44.851709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:44.851741 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:44.851901 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.851914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.853415 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.853421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.855233 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.855252 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.856924 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.856930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.857135 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.857172 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:33:44.857278 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 11:33:44.857429 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10627,"thread_start_us":14758,"threads_started":1}
I20260929 11:33:44.857800 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.837674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.6.9.195:43307)
0929 11:33:44.851363 (+ 13689us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:44.851368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:44.851388 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:33:44.851459 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:44.851464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:44.851534 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:44.851650 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:44.852104 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.853234 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.855463 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.856747 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.856879 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.856989 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:44.857334 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 11:33:44.857488 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13568,"thread_start_us":13468,"threads_started":1}
I20260929 11:33:44.858767 13663 tablet_bootstrap.cc:654] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:44.858702 13492 tablet_service.cc:1467] Processing CreateTablet for tablet b091332db96b40f7895fd46f1b9408c0 (DEFAULT_TABLE table=test-table [id=e62516a205e94ee8ac56932a88eae12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:44.859157 13492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b091332db96b40f7895fd46f1b9408c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:44.859256 13664 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Bootstrap starting.
I20260929 11:33:44.861114 13664 tablet_bootstrap.cc:654] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:44.862761 13663 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: No bootstrap required, opened a new log
I20260929 11:33:44.862936 13663 ts_tablet_manager.cc:1397] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 11:33:44.863225 13664 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: No bootstrap required, opened a new log
I20260929 11:33:44.863363 13664 ts_tablet_manager.cc:1397] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:33:44.863794 13663 raft_consensus.cc:348] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.864049 13663 raft_consensus.cc:374] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:44.864128 13663 raft_consensus.cc:729] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a21eea0381e4479943d18ef9d9dc7c8, State: Initialized, Role: FOLLOWER
I20260929 11:33:44.863991 13664 raft_consensus.cc:348] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.864235 13664 raft_consensus.cc:374] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:44.864318 13664 raft_consensus.cc:729] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98d47edc02a42d59ef41d5553f37fb1, State: Initialized, Role: FOLLOWER
I20260929 11:33:44.864416 13663 consensus_queue.cc:260] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.864580 13664 consensus_queue.cc:260] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.869557 13669 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Bootstrap starting.
I20260929 11:33:44.871529 13663 ts_tablet_manager.cc:1428] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 11:33:44.873150 13664 ts_tablet_manager.cc:1428] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260929 11:33:44.875216 13266 heartbeater.cc:503] Master 127.6.9.252:37129 was elected leader, sending a full tablet report...
I20260929 11:33:44.876879 13417 heartbeater.cc:503] Master 127.6.9.252:37129 was elected leader, sending a full tablet report...
I20260929 11:33:44.877305 13669 tablet_bootstrap.cc:654] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:44.879189 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.871893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:57391)
0929 11:33:44.878789 (+ 6896us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.878937 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6748,"thread_start_us":6915,"threads_started":1}
I20260929 11:33:44.880192 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.879814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:48741)
0929 11:33:44.879975 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.880056 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 11:33:44.880245 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.873355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:35003)
0929 11:33:44.879833 (+ 6478us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.879969 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2482,"thread_start_us":2499,"threads_started":1}
I20260929 11:33:44.880501 13669 tablet_bootstrap.cc:492] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: No bootstrap required, opened a new log
I20260929 11:33:44.880652 13669 ts_tablet_manager.cc:1397] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 11:33:44.881461 13669 raft_consensus.cc:348] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.881716 13669 raft_consensus.cc:374] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:44.881821 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.880857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:36289)
0929 11:33:44.881398 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.881540 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":272,"threads_started":1}
I20260929 11:33:44.881815 13669 raft_consensus.cc:729] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef156af6f84be5957cefb702865298, State: Initialized, Role: FOLLOWER
I20260929 11:33:44.882056 13669 consensus_queue.cc:260] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.883813 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.882977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:44735)
0929 11:33:44.883418 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.883535 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260929 11:33:44.882741 13669 ts_tablet_manager.cc:1428] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:44.884701 13558 heartbeater.cc:503] Master 127.6.9.252:37129 was elected leader, sending a full tablet report...
I20260929 11:33:44.885206 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.884512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:44129)
0929 11:33:44.884984 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.885065 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260929 11:33:44.978124 13668 raft_consensus.cc:482] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:44.978379 13668 raft_consensus.cc:504] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.980047 13668 leader_election.cc:288] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045), 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:44.987537 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.980422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:48573 (local address 127.6.9.195:43307)
0929 11:33:44.980619 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:44.980640 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:44.980748 (+ 108us) server_negotiation.cc:407] Connection header received
0929 11:33:44.980934 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:44.980939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:44.981016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:44.981113 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:44.981890 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.982836 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.985113 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.986349 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.986482 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.986591 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:44.987130 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 11:33:44.987269 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:33:44.987532 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.980438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:41675 (local address 127.6.9.193:42045)
0929 11:33:44.980613 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:44.980619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:44.980812 (+ 193us) server_negotiation.cc:407] Connection header received
0929 11:33:44.980931 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:44.980936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:44.981016 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:44.981113 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:44.981553 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.982547 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.985113 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.986349 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.986482 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.986591 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:44.987136 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 11:33:44.987269 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 11:33:44.987624 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.980277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:42045 (local address 127.6.9.194:41675)
0929 11:33:44.980716 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.980731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:44.980816 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:44.981147 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:44.981152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:44.981179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:44.981426 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.981434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.982700 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.982704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.984900 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.984917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.986618 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.986624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.986850 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.986942 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 11:33:44.987081 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 11:33:44.987322 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":312}
I20260929 11:33:44.988004 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.980278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:43307 (local address 127.6.9.194:48573)
0929 11:33:44.980455 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.980649 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:44.980814 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:44.981557 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:44.981559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:44.981577 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:44.981752 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.981758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.982964 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.982969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.984942 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:44.984951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:44.986592 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:44.986602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:44.986848 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:44.986942 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 11:33:44.987099 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 11:33:44.987322 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":146}
I20260929 11:33:44.988286 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091332db96b40f7895fd46f1b9408c0" candidate_uuid: "b98d47edc02a42d59ef41d5553f37fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" is_pre_election: true
I20260929 11:33:44.988554 13220 raft_consensus.cc:2454] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98d47edc02a42d59ef41d5553f37fb1 in term 0.
I20260929 11:33:44.988832 13512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091332db96b40f7895fd46f1b9408c0" candidate_uuid: "b98d47edc02a42d59ef41d5553f37fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef156af6f84be5957cefb702865298" is_pre_election: true
I20260929 11:33:44.989084 13304 leader_election.cc:302] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a21eea0381e4479943d18ef9d9dc7c8, b98d47edc02a42d59ef41d5553f37fb1; no voters:
I20260929 11:33:44.989289 13512 raft_consensus.cc:2454] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98d47edc02a42d59ef41d5553f37fb1 in term 0.
I20260929 11:33:44.989557 13668 raft_consensus.cc:2781] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:44.989697 13668 raft_consensus.cc:482] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:44.989796 13668 raft_consensus.cc:3037] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:44.991950 13668 raft_consensus.cc:504] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.992506 13668 leader_election.cc:288] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [CANDIDATE]: Term 1 election: Requested vote from peers 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045), 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307)
I20260929 11:33:44.992981 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091332db96b40f7895fd46f1b9408c0" candidate_uuid: "b98d47edc02a42d59ef41d5553f37fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a21eea0381e4479943d18ef9d9dc7c8"
I20260929 11:33:44.993054 13512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b091332db96b40f7895fd46f1b9408c0" candidate_uuid: "b98d47edc02a42d59ef41d5553f37fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef156af6f84be5957cefb702865298"
I20260929 11:33:44.993161 13220 raft_consensus.cc:3037] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:44.993255 13512 raft_consensus.cc:3037] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:44.995076 13220 raft_consensus.cc:2454] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98d47edc02a42d59ef41d5553f37fb1 in term 1.
I20260929 11:33:44.995321 13512 raft_consensus.cc:2454] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98d47edc02a42d59ef41d5553f37fb1 in term 1.
I20260929 11:33:44.995457 13304 leader_election.cc:302] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a21eea0381e4479943d18ef9d9dc7c8, b98d47edc02a42d59ef41d5553f37fb1; no voters:
I20260929 11:33:44.995787 13668 raft_consensus.cc:2781] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:44.996222 13668 raft_consensus.cc:686] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 LEADER]: Becoming Leader. State: Replica: b98d47edc02a42d59ef41d5553f37fb1, State: Running, Role: LEADER
I20260929 11:33:44.996582 13668 consensus_queue.cc:237] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } }
I20260929 11:33:44.999668 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:44.999226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:56089)
0929 11:33:44.999381 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:44.999485 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 11:33:45.000313 13087 catalog_manager.cc:5495] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "b98d47edc02a42d59ef41d5553f37fb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98d47edc02a42d59ef41d5553f37fb1" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:45.001278 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.000350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:33971)
0929 11:33:45.000654 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.001136 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260929 11:33:45.019001 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.014249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.6.9.194:33983)
0929 11:33:45.014410 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:45.014414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:45.014432 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:45.014561 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:45.014565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:45.014655 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:45.014790 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:45.015101 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.015788 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.017081 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.017496 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.017586 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:45.017928 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:45.018499 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:45.018606 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:45.018717 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 11:33:45.018790 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:33:45.023075 13220 raft_consensus.cc:1261] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Refusing update from remote peer b98d47edc02a42d59ef41d5553f37fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:45.023373 13512 raft_consensus.cc:1261] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Refusing update from remote peer b98d47edc02a42d59ef41d5553f37fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:45.023764 13676 consensus_queue.cc:1035] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a21eea0381e4479943d18ef9d9dc7c8" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:45.024192 13668 consensus_queue.cc:1035] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ef156af6f84be5957cefb702865298" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 43307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:45.026906 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.026299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:34063)
0929 11:33:45.026550 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.026662 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 11:33:45.026894 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.026388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:36401)
0929 11:33:45.026585 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.026689 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 11:33:45.033545 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.033028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.193:42051)
0929 11:33:45.033290 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.033380 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260929 11:33:45.041864 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.041229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:58327)
0929 11:33:45.041546 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.041646 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 11:33:45.043349 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.042948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.194:38761)
0929 11:33:45.043103 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.043203 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 11:33:45.044394 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.043888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.6.9.195:39377)
0929 11:33:45.044098 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.044205 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 11:33:45.045463 13681 mvcc.cc:205] Tried to move back new op lower bound from 7334631936087531520 to 7334631935989563392. Current Snapshot: MvccSnapshot[applied={T|T < 7334631936087531520}]
I20260929 11:33:45.054383 13088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:45.054669 13088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:45.060204 13088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91dee2623f154dd1b596f7625f8641ee: Sending DeleteTablet for 3 replicas of tablet b091332db96b40f7895fd46f1b9408c0
I20260929 11:33:45.061244 13351 tablet_service.cc:1514] Processing DeleteTablet for tablet b091332db96b40f7895fd46f1b9408c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:45 UTC) from {username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:44584
I20260929 11:33:45.061435 13200 tablet_service.cc:1514] Processing DeleteTablet for tablet b091332db96b40f7895fd46f1b9408c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:45 UTC) from {username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:58886
I20260929 11:33:45.062129 13492 tablet_service.cc:1514] Processing DeleteTablet for tablet b091332db96b40f7895fd46f1b9408c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:45 UTC) from {username='oryx', principal='oryx/127.6.9.252@KRBTEST.COM'} at 127.0.0.1:54200
I20260929 11:33:45.062690 13689 tablet_replica.cc:330] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: stopping tablet replica
I20260929 11:33:45.062983 13689 raft_consensus.cc:2229] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:45.063334 13689 raft_consensus.cc:2258] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:45.063407 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13060
I20260929 11:33:45.064906 13689 ts_tablet_manager.cc:1905] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:45.071544 13689 ts_tablet_manager.cc:1918] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:45.071774 13689 log.cc:1196] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/b091332db96b40f7895fd46f1b9408c0
I20260929 11:33:45.072294 13689 ts_tablet_manager.cc:1939] T b091332db96b40f7895fd46f1b9408c0 P 8a21eea0381e4479943d18ef9d9dc7c8: Deleting consensus metadata
I20260929 11:33:45.073719 13072 catalog_manager.cc:4841] TS 8a21eea0381e4479943d18ef9d9dc7c8 (127.6.9.193:42045): tablet b091332db96b40f7895fd46f1b9408c0 (table test-table [id=e62516a205e94ee8ac56932a88eae12e]) successfully deleted
I20260929 11:33:45.073918 13690 tablet_replica.cc:330] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: stopping tablet replica
I20260929 11:33:45.074116 13690 raft_consensus.cc:2229] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:45.074427 13690 raft_consensus.cc:2258] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:45.075404 13690 ts_tablet_manager.cc:1905] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:45.078053 13688 tablet_replica.cc:330] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: stopping tablet replica
I20260929 11:33:45.078272 13688 raft_consensus.cc:2229] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:45.078943 13688 raft_consensus.cc:2258] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:45.080576 13688 ts_tablet_manager.cc:1905] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:45.080721 13690 ts_tablet_manager.cc:1918] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:45.080870 13690 log.cc:1196] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/b091332db96b40f7895fd46f1b9408c0
I20260929 11:33:45.081347 13690 ts_tablet_manager.cc:1939] T b091332db96b40f7895fd46f1b9408c0 P 36ef156af6f84be5957cefb702865298: Deleting consensus metadata
I20260929 11:33:45.082472 13075 catalog_manager.cc:4841] TS 36ef156af6f84be5957cefb702865298 (127.6.9.195:43307): tablet b091332db96b40f7895fd46f1b9408c0 (table test-table [id=e62516a205e94ee8ac56932a88eae12e]) successfully deleted
I20260929 11:33:45.087568 13688 ts_tablet_manager.cc:1918] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:45.087740 13688 log.cc:1196] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/b091332db96b40f7895fd46f1b9408c0
I20260929 11:33:45.088199 13688 ts_tablet_manager.cc:1939] T b091332db96b40f7895fd46f1b9408c0 P b98d47edc02a42d59ef41d5553f37fb1: Deleting consensus metadata
I20260929 11:33:45.089429 13072 catalog_manager.cc:4841] TS b98d47edc02a42d59ef41d5553f37fb1 (127.6.9.194:33983): tablet b091332db96b40f7895fd46f1b9408c0 (table test-table [id=e62516a205e94ee8ac56932a88eae12e]) successfully deleted
I20260929 11:33:45.091646 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.064262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.6.9.193:42045)
0929 11:33:45.064404 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:45.064410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:45.072714 (+ 8304us) server_negotiation.cc:407] Connection header received
0929 11:33:45.073120 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:45.073128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:45.073236 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:45.073311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:45.079038 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.080171 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.082318 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.082842 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.082923 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:45.085709 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:45.085725 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:45.085741 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:45.085784 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:45.088741 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.089515 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.089523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.089526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.089693 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.090099 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.090104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.090108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.090494 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:45.090774 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:45.091097 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:33:45.091309 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 11:33:45.092319 13260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:45.207579 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13060
I20260929 11:33:45.222527 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13282
I20260929 11:33:45.237175 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.223292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.6.9.194:33983)
0929 11:33:45.223465 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:45.223469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:45.223546 (+ 77us) server_negotiation.cc:407] Connection header received
0929 11:33:45.223739 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:45.223743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:45.223848 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:45.223935 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:45.225718 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.226480 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.228263 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.228649 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.228804 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:45.231261 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:45.231276 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:45.231293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:45.231337 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:45.234291 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.234982 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.234990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.234992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.235147 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.235514 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.235518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.235520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.235919 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:45.236218 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:45.236548 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 11:33:45.236812 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:33:45.237860 13411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:45.324955 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.324412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60690)
0929 11:33:45.324633 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.324773 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 11:33:45.326172 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.325832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46741 (local address 127.0.0.1:60692)
0929 11:33:45.325996 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:45.326062 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:33:45.355329 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13282
I20260929 11:33:45.370576 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13429
I20260929 11:33:45.383808 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:45.371314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.6.9.195:43307)
0929 11:33:45.371478 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:45.371482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:45.371500 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:45.371645 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:45.371649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:45.371721 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:45.371804 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:45.373193 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.373957 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.375126 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:45.375534 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:45.375639 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:45.378081 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:45.378097 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:45.378114 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:45.378147 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:45.381117 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.381732 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.381740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.381743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.381882 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:45.382295 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:45.382298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:45.382301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:45.382679 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:45.382959 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:45.383278 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 11:33:45.383486 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:33:45.384469 13552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:45.508169 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13429
I20260929 11:33:45.525701 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12763
I20260929 11:33:45.526834 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:45.643111 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12763
I20260929 11:33:45.659659 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 12863
I20260929 11:33:45.660769 13118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:45.781656 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 12863
2026-09-29T11:33:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11629 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:45 dist-test-slave-c0l9 krb5kdc[13700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:45 dist-test-slave-c0l9 krb5kdc[13700](info): set up 2 sockets
Sep 29 11:33:45 dist-test-slave-c0l9 krb5kdc[13700](info): commencing operation
krb5kdc: starting...
W20260929 11:33:46.797382 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:46 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab.
Entry for principal oryx/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab.
I20260929 11:33:47.003767 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:32947
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab
--principal=oryx/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:47.322415 13713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:47.322736 13713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:47.322870 13713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:47.330180 13713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:47.330319 13713 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:33:47.330399 13713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:47.330483 13713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:47.330549 13713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:47.330631 13713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:47.340796 13713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:32947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.254.keytab
--principal=oryx/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:47.342092 13713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:47.343428 13713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:47.355854 13718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:47.355749 13719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:47.355726 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:47.356909 13713 server_base.cc:1034] running on GCE node
I20260929 11:33:47.357897 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:47.359627 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:47.360874 13713 hybrid_clock.cc:648] HybridClock initialized: now 1790681627360805 us; error 45 us; skew 500 ppm
I20260929 11:33:47.361287 13713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:47 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681627, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:47.369257 13713 init.cc:377] Logged in from keytab as oryx/127.6.9.254@KRBTEST.COM (short username oryx)
I20260929 11:33:47.370862 13713 webserver.cc:460] Webserver started at http://127.6.9.254:39933/ using document root <none> and password file <none>
I20260929 11:33:47.371670 13713 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:47.371780 13713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:47.372125 13713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:47.375367 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "4f6b95a4c9674b13a045e105a740585c"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "0f70ebc082647b408daabc86e40db04e"
server_key_iv: "399ecd26a1f7d26c56b34646e99b59fe"
server_key_version: "encryptionkey@0"
I20260929 11:33:47.376237 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "4f6b95a4c9674b13a045e105a740585c"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "0f70ebc082647b408daabc86e40db04e"
server_key_iv: "399ecd26a1f7d26c56b34646e99b59fe"
server_key_version: "encryptionkey@0"
I20260929 11:33:47.381719 13713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:33:47.385365 13728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:47.386775 13713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:47.386952 13713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "4f6b95a4c9674b13a045e105a740585c"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "0f70ebc082647b408daabc86e40db04e"
server_key_iv: "399ecd26a1f7d26c56b34646e99b59fe"
server_key_version: "encryptionkey@0"
I20260929 11:33:47.387171 13713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:47.406504 13713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:47.411149 13713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:47.411429 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:47.432992 13713 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:32947
I20260929 11:33:47.433023 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:32947 every 8 connection(s)
I20260929 11:33:47.434789 13713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:47.439415 13790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:47.442860 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13713
I20260929 11:33:47.443061 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:47.443694 6183 external_mini_cluster.cc:1428] Setting key 255ac1eaa84e516aa78096acce279a64
I20260929 11:33:47.453708 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Bootstrap starting.
I20260929 11:33:47.457458 13790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:47.458487 13790 log.cc:824] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:33:47 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:47.461547 13790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: No bootstrap required, opened a new log
I20260929 11:33:47.468266 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:47.445954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.6.9.254:32947)
0929 11:33:47.446792 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:47.446810 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:47.446860 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:33:47.449163 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:47.449205 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:47.449854 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:47.450384 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:47.451857 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:47.453889 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:47.455277 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:47.456353 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:47.456547 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:47.460641 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:47.460650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:47.460701 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:47.460756 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:47.463810 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:47.464602 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:47.464610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:47.464617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:47.464846 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:47.465236 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:47.465241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:47.465243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:47.465846 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:47.466149 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:47.466627 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 11:33:47.467097 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":338,"threads_started":1}
I20260929 11:33:47.469607 13790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } }
I20260929 11:33:47.469997 13790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:47.470103 13790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6b95a4c9674b13a045e105a740585c, State: Initialized, Role: FOLLOWER
I20260929 11:33:47.470655 13790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } }
I20260929 11:33:47.470897 13790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:47.471028 13790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:47.471232 13790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:47.473451 13790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } }
I20260929 11:33:47.473935 13790 leader_election.cc:302] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f6b95a4c9674b13a045e105a740585c; no voters:
I20260929 11:33:47.474463 13790 leader_election.cc:288] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:47.474601 13795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:47.475252 13795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [term 1 LEADER]: Becoming Leader. State: Replica: 4f6b95a4c9674b13a045e105a740585c, State: Running, Role: LEADER
I20260929 11:33:47.475808 13795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } }
I20260929 11:33:47.476428 13790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:47.478564 13797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f6b95a4c9674b13a045e105a740585c. Latest consensus state: current_term: 1 leader_uuid: "4f6b95a4c9674b13a045e105a740585c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } } }
I20260929 11:33:47.479131 13797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:47.479049 13796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f6b95a4c9674b13a045e105a740585c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6b95a4c9674b13a045e105a740585c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 32947 } } }
I20260929 11:33:47.479311 13796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:47.479982 13804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:47.485888 13804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:47.495508 13804 catalog_manager.cc:1347] Generated new cluster ID: 67f8a975420a4d2999ac782c168c5554
I20260929 11:33:47.495730 13804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:47.523775 13804 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:47.525220 13804 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:47.534998 13804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Generated new TSK 0
I20260929 11:33:47.535869 13804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
I20260929 11:33:47.594295 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32947
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:47.955296 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:47.955608 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:47.955739 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:47.963339 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:47.963470 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:47.963583 13816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:47.963676 13816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:47.973897 13816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:47.975176 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:47.976563 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:47.989476 13821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:47.989455 13822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:47.989411 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:47.990365 13816 server_base.cc:1034] running on GCE node
I20260929 11:33:47.990998 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:47.992019 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:47.993232 13816 hybrid_clock.cc:648] HybridClock initialized: now 1790681627993153 us; error 65 us; skew 500 ppm
I20260929 11:33:47.993543 13816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:47 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681627, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:47.996654 13816 init.cc:377] Logged in from keytab as oryx/127.6.9.193@KRBTEST.COM (short username oryx)
I20260929 11:33:47.998224 13816 webserver.cc:460] Webserver started at http://127.6.9.193:42619/ using document root <none> and password file <none>
I20260929 11:33:47.998929 13816 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:47.999033 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:47.999351 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:48.002432 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "4603af21508c6b0060f58856e88388ba"
server_key_iv: "3b92045240a5628d9017843502491f22"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.003211 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "4603af21508c6b0060f58856e88388ba"
server_key_iv: "3b92045240a5628d9017843502491f22"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.008368 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 11:33:48.011942 13831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:48.013422 13816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:48.013589 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
format_stamp: "Formatted at 2026-09-29 11:33:47 on dist-test-slave-c0l9"
server_key: "4603af21508c6b0060f58856e88388ba"
server_key_iv: "3b92045240a5628d9017843502491f22"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.013813 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:48.034000 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:48.038098 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:48.038369 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:48.041149 13816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:48.041267 13816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:48.041402 13816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:48.041483 13816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:33:48 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681627, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:48.063766 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.041634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:59279 (local address 127.6.9.254:32947)
0929 11:33:48.042367 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:48.042371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:48.043418 (+ 1047us) server_negotiation.cc:407] Connection header received
0929 11:33:48.045532 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:48.045538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:48.045622 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:48.045728 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:48.047929 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.048965 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.051421 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.051874 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.052009 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.056896 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:48.056901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:48.056905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:48.056939 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:48.060201 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.061228 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.061234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.061238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.061419 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.061868 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.061873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.061877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.062244 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:48.062402 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:48.063069 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 11:33:48.063333 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":531,"threads_started":1}
I20260929 11:33:48.064584 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.042381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.193:59279)
0929 11:33:48.043260 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:48.043301 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:48.045237 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:48.045982 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:48.045990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:48.046788 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:48.047663 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.047693 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.049132 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.049137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.051254 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.051274 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.052187 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.052196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.052512 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.053342 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:48.053378 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:48.056620 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:48.060361 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.060371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.060395 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.061077 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.061541 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.061544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.061547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.061762 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.062353 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:48.062359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:48.062718 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 11:33:48.062941 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 11:33:48.063339 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":351,"threads_started":1}
I20260929 11:33:48.088871 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.071127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:54197 (local address 127.6.9.254:32947)
0929 11:33:48.071352 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:48.071360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:48.073508 (+ 2148us) server_negotiation.cc:407] Connection header received
0929 11:33:48.073835 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:48.073842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:48.073947 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:48.074056 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:48.075606 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.076645 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.078776 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.079430 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.079578 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.081784 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:48.081792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:48.081798 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:48.081846 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:48.085356 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.086228 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.086235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.086238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.086413 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.086952 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.086958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.086962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.087387 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:48.087549 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:48.088244 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 11:33:48.088514 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:33:48.089138 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.070618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.193:54197)
0929 11:33:48.073342 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:48.073383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:48.073662 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:48.074218 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:48.074222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:48.074907 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:48.075437 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.075452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.076794 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.076803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.078587 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.078609 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.079625 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.079629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.079769 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.080410 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:48.080436 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:48.081558 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:48.085514 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.085520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.085524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.086088 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.086565 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.086571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.086576 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.086792 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.087624 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:48.087630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:48.087905 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 11:33:48.088098 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 11:33:48.088529 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2535,"thread_start_us":2293,"threads_started":1}
I20260929 11:33:48.095302 13816 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:43551
I20260929 11:33:48.095324 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:43551 every 8 connection(s)
I20260929 11:33:48.096557 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:48.098693 13952 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32947
I20260929 11:33:48.098929 13952 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:48.099503 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13816
I20260929 11:33:48.099587 13952 heartbeater.cc:511] Master 127.6.9.254:32947 requested a full tablet report, sending...
I20260929 11:33:48.099673 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:48.100292 6183 external_mini_cluster.cc:1428] Setting key 6c29850b7aa6412a4adfa27cc2a9a290
I20260929 11:33:48.101948 13745 ts_manager.cc:194] Registered new tserver with Master: 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551)
I20260929 11:33:48.103840 13745 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:59279
WARNING: no policy specified for oryx/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
I20260929 11:33:48.146201 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32947
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:48.469271 13956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:48.469563 13956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:48.469660 13956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:48.476994 13956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:48.477120 13956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:48.477228 13956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:48.477329 13956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:48.487435 13956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:48.488739 13956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:48.490087 13956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:48.503527 13961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:48.503544 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:48.503571 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:48.504338 13956 server_base.cc:1034] running on GCE node
I20260929 11:33:48.505090 13956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:48.506109 13956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:48.507367 13956 hybrid_clock.cc:648] HybridClock initialized: now 1790681628507289 us; error 49 us; skew 500 ppm
I20260929 11:33:48.507725 13956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:48 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681628, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:48.510994 13956 init.cc:377] Logged in from keytab as oryx/127.6.9.194@KRBTEST.COM (short username oryx)
I20260929 11:33:48.512540 13956 webserver.cc:460] Webserver started at http://127.6.9.194:42137/ using document root <none> and password file <none>
I20260929 11:33:48.513311 13956 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:48.513405 13956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:48.513815 13956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:48.516995 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "f41aafeadae0448c80e82cd5a05d7837"
format_stamp: "Formatted at 2026-09-29 11:33:48 on dist-test-slave-c0l9"
server_key: "1e599056f66957082e3952ec155269f5"
server_key_iv: "dff3d447884d93386fbcc225642ac1b9"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.517863 13956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "f41aafeadae0448c80e82cd5a05d7837"
format_stamp: "Formatted at 2026-09-29 11:33:48 on dist-test-slave-c0l9"
server_key: "1e599056f66957082e3952ec155269f5"
server_key_iv: "dff3d447884d93386fbcc225642ac1b9"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.523198 13956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:48.526968 13971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:48.528373 13956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:33:48.528532 13956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "f41aafeadae0448c80e82cd5a05d7837"
format_stamp: "Formatted at 2026-09-29 11:33:48 on dist-test-slave-c0l9"
server_key: "1e599056f66957082e3952ec155269f5"
server_key_iv: "dff3d447884d93386fbcc225642ac1b9"
server_key_version: "encryptionkey@0"
I20260929 11:33:48.528818 13956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:48.556711 13956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:48.560956 13956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:48.561192 13956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:48.563879 13956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:48.563977 13956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:48.564116 13956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:48.564182 13956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:33:48 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681628, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:48.588420 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.564792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:43167 (local address 127.6.9.254:32947)
0929 11:33:48.564948 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:48.564952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:48.566603 (+ 1651us) server_negotiation.cc:407] Connection header received
0929 11:33:48.568944 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:48.568951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:48.569043 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:48.569109 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:48.571661 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.572653 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.575678 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.576306 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.576462 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.581642 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:48.581647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:48.581651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:48.581686 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:48.584235 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.585323 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.585329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.585333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.585469 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.586184 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.586190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.586201 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.586531 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:48.586680 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:48.587708 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 11:33:48.587989 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 11:33:48.589438 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.565532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.194:43167)
0929 11:33:48.566419 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:48.566463 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:48.568503 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:48.569653 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:48.569665 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:48.570511 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:48.571428 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.571467 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.572870 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.572876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.575452 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.575474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.576597 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.576607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.576938 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.577849 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:48.577886 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:48.581393 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:48.584380 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.584390 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.584411 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.585172 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.585582 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.585588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.585592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.586050 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.586670 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:48.586678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:48.587183 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 11:33:48.587561 (+ 378us) client_negotiation.cc:240] Negotiation successful
0929 11:33:48.588058 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":380,"threads_started":1}
I20260929 11:33:48.609145 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.595514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.194:34879)
0929 11:33:48.596110 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:48.596129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:48.596275 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:48.596582 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:48.596586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:48.597186 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:48.597668 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.597685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.599039 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.599042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.600084 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:48.600095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.600848 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.600855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:48.601035 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.601709 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:48.601727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:48.602554 (+ 827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:48.606398 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.606404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.606408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.606946 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.607408 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:48.607413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:48.607416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:48.607541 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:48.608249 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:48.608253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:48.608411 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:33:48.608523 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:33:48.608821 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":321,"threads_started":1}
I20260929 11:33:48.609279 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:48.595434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34879 (local address 127.6.9.254:32947)
0929 11:33:48.595597 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:48.595602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:48.596212 (+ 610us) server_negotiation.cc:407] Connection header received
0929 11:33:48.596396 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:48.596401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:48.596465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:48.596544 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:48.597881 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.598860 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.600224 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:48.600627 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:48.600785 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:48.602795 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:48.602802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:48.602807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:48.602850 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:48.606234 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.607093 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.607099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.607102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.607283 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:48.607639 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:48.607643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:48.607647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:48.608129 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:48.608299 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:48.608632 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 11:33:48.608913 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 11:33:48.616150 13956 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:36373
I20260929 11:33:48.616170 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:36373 every 8 connection(s)
I20260929 11:33:48.617427 13956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:48.619688 14091 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32947
I20260929 11:33:48.619917 14091 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:48.620564 14091 heartbeater.cc:511] Master 127.6.9.254:32947 requested a full tablet report, sending...
I20260929 11:33:48.622007 13745 ts_manager.cc:194] Registered new tserver with Master: f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373)
I20260929 11:33:48.622838 13745 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:43167
I20260929 11:33:48.626876 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 13956
I20260929 11:33:48.627023 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:48.627424 6183 external_mini_cluster.cc:1428] Setting key 3473ba7cdc437d22041378c63f7843df
WARNING: no policy specified for oryx/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
I20260929 11:33:48.671731 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:32947
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:49.002460 14095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:49.002776 14095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:49.002921 14095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:49.010617 14095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:49.010746 14095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:49.010835 14095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:33:49.010915 14095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:49.020900 14095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.9.254:32947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:49.022184 14095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:49.023511 14095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:49.036255 14100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:49.036295 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:49.036258 14103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:49.037395 14095 server_base.cc:1034] running on GCE node
I20260929 11:33:49.037953 14095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:49.038851 14095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:49.040068 14095 hybrid_clock.cc:648] HybridClock initialized: now 1790681629040005 us; error 43 us; skew 500 ppm
I20260929 11:33:49.040365 14095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681629, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:49.043467 14095 init.cc:377] Logged in from keytab as oryx/127.6.9.195@KRBTEST.COM (short username oryx)
I20260929 11:33:49.045006 14095 webserver.cc:460] Webserver started at http://127.6.9.195:33085/ using document root <none> and password file <none>
I20260929 11:33:49.045693 14095 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:49.045799 14095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:49.046123 14095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:49.049129 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "f378f76b47734b28a5d5b5e3c9424f10"
format_stamp: "Formatted at 2026-09-29 11:33:49 on dist-test-slave-c0l9"
server_key: "35160b8ce5eda64e6f335ecc8c149baf"
server_key_iv: "6970d8ce226e95807576670e9e61272d"
server_key_version: "encryptionkey@0"
I20260929 11:33:49.049926 14095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "f378f76b47734b28a5d5b5e3c9424f10"
format_stamp: "Formatted at 2026-09-29 11:33:49 on dist-test-slave-c0l9"
server_key: "35160b8ce5eda64e6f335ecc8c149baf"
server_key_iv: "6970d8ce226e95807576670e9e61272d"
server_key_version: "encryptionkey@0"
I20260929 11:33:49.055095 14095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:49.058696 14110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:49.060036 14095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:49.060196 14095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "f378f76b47734b28a5d5b5e3c9424f10"
format_stamp: "Formatted at 2026-09-29 11:33:49 on dist-test-slave-c0l9"
server_key: "35160b8ce5eda64e6f335ecc8c149baf"
server_key_iv: "6970d8ce226e95807576670e9e61272d"
server_key_version: "encryptionkey@0"
I20260929 11:33:49.060421 14095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:49.091383 14095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:49.095978 14095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:49.096246 14095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:49.100710 14095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:49.100878 14095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 11:33:49.101018 14095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:49.101111 14095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:49.106819 13952 heartbeater.cc:503] Master 127.6.9.254:32947 was elected leader, sending a full tablet report...
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681629, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:49.124253 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.101438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.195:50353)
0929 11:33:49.102248 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.102286 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.104225 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.104957 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.104965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.105726 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:49.106594 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.106616 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.107994 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.107999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.109843 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.109858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.110869 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.110875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.111100 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.111819 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:49.111843 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:49.114960 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:49.118713 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.118723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.118742 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.119407 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.119924 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.119929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.119933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.120190 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.120825 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:49.120837 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:49.121439 (+ 602us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.121855 (+ 416us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.122579 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":334,"threads_started":1}
I20260929 11:33:49.124305 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.100750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:50353 (local address 127.6.9.254:32947)
0929 11:33:49.100906 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.100911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.102418 (+ 1507us) server_negotiation.cc:407] Connection header received
0929 11:33:49.104535 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.104541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.104638 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.104773 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:49.106780 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.107792 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.110000 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.110535 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.110669 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.115225 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:49.115231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:49.115234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:49.115271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:49.118562 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.119613 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.119619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.119622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.119807 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.120328 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.120334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.120338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.120656 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:49.120840 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.123608 (+ 2768us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.123909 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:33:49.140946 14095 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:34747
I20260929 11:33:49.140959 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:34747 every 8 connection(s)
I20260929 11:33:49.142545 14095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:49.144831 14230 heartbeater.cc:347] Connected to a master server at 127.6.9.254:32947
I20260929 11:33:49.145063 14230 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:49.145748 14230 heartbeater.cc:511] Master 127.6.9.254:32947 requested a full tablet report, sending...
I20260929 11:33:49.145781 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.130878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:56301 (local address 127.6.9.254:32947)
0929 11:33:49.131079 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.131086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.131763 (+ 677us) server_negotiation.cc:407] Connection header received
0929 11:33:49.131976 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.131982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.132068 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.132185 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:49.133270 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.134432 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.136238 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.136946 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.137081 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.139351 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:49.139358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:49.139362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:49.139402 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:49.142820 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.143670 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.143675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.143678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.143859 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.144260 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.144265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.144269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.144616 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:49.144799 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.145217 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.145432 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:33:49.145917 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.130904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.6.9.195:56301)
0929 11:33:49.131639 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.131660 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.131830 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.132199 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.132211 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.132752 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:49.133104 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.133118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.134576 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.134582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.136070 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.136097 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.137162 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.137169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.137362 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.138055 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:49.138077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:49.139144 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:49.142978 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.142982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.142985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.143491 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.143981 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.143986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.143990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.144149 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.144761 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:49.144766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:49.144963 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.145118 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.145470 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":351,"threads_started":1}
I20260929 11:33:49.147141 13744 ts_manager.cc:194] Registered new tserver with Master: f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:49.148002 13744 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:50353
I20260929 11:33:49.152621 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14095
I20260929 11:33:49.152803 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:49.153182 6183 external_mini_cluster.cc:1428] Setting key 1f3c21a6cfc78c64451974e6a63eb185
I20260929 11:33:49.156028 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:33:49.203495 13840 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:49.193507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.6.9.254:32947)
0929 11:33:49.193735 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.193742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.193933 (+ 191us) server_negotiation.cc:407] Connection header received
0929 11:33:49.194117 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.194123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.194185 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.194267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:49.195631 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.196346 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.197591 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.198019 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.198131 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.203186 (+ 5055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53362: BlockingRecv error: recv got EOF from 127.0.0.1:53362 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681629, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:49.219103 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.206258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.6.9.254:32947)
0929 11:33:49.206436 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.206442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.206703 (+ 261us) server_negotiation.cc:407] Connection header received
0929 11:33:49.206862 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.206867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.206936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.207017 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:49.208091 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.208855 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.210122 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.210531 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.210640 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.214229 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:49.214234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:49.214237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:49.214271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:49.216491 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.217225 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.217231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.217235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.217362 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.217711 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.217715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.217717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.218008 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:49.218299 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.218584 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.218811 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:33:49.222924 13744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:49.229781 13744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:49.256987 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.241831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36373 (local address 127.0.0.1:60740)
0929 11:33:49.243601 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.243663 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.243877 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.246104 (+ 2227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.246113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.246145 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.246543 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.246554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.249737 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.249743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.252361 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.252383 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.255864 (+ 3481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.255871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.256133 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.256197 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.256348 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.256562 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":2660,"threads_started":1}
I20260929 11:33:49.258015 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.242004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.6.9.194:36373)
0929 11:33:49.243001 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.243012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.244079 (+ 1067us) server_negotiation.cc:407] Connection header received
0929 11:33:49.244188 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.244197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.244479 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.245834 (+ 1355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.246770 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.249566 (+ 2796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.253910 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.255376 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.255502 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.255720 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.257271 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.257463 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":712,"threads_started":1}
I20260929 11:33:49.262503 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.242568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.6.9.193:43551)
0929 11:33:49.254983 (+ 12415us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.254992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.255023 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:33:49.255124 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.255132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.255442 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.256571 (+ 1129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.256822 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.258217 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.260311 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.261408 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.261539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.261736 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.262019 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.262192 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12223,"thread_start_us":13032,"threads_started":1}
I20260929 11:33:49.262537 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.241999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43551 (local address 127.0.0.1:39978)
0929 11:33:49.244761 (+ 2762us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.244782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.244884 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.255874 (+ 10990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.255879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.255907 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.256369 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.256382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.258388 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.258393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.260151 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.260163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.261631 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.261638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.261824 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.261862 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.261967 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.262115 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2633,"mutex_wait_us":57,"thread_start_us":1162,"threads_started":1}
I20260929 11:33:49.262120 14021 tablet_service.cc:1467] Processing CreateTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (DEFAULT_TABLE table=test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.263998 14021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.266002 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.245117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.0.0.1:59250)
0929 11:33:49.257013 (+ 11896us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.257039 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.257164 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.258845 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.258850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.258879 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.259272 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.259285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.261246 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.261252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.263243 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.263257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.265174 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.265180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.265356 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.265389 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.265488 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.265625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11732,"mutex_wait_us":3,"thread_start_us":12115,"threads_started":1}
I20260929 11:33:49.266044 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.245256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.6.9.195:34747)
0929 11:33:49.257770 (+ 12514us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.257779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.257815 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:33:49.257915 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.257924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.258207 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.258421 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.259494 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.261079 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.263397 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.264690 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.264832 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.265044 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.265558 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.265731 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12358,"thread_start_us":13391,"threads_started":1}
I20260929 11:33:49.266718 13882 tablet_service.cc:1467] Processing CreateTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (DEFAULT_TABLE table=test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.268090 13882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.271118 14160 tablet_service.cc:1467] Processing CreateTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (DEFAULT_TABLE table=test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.272514 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.278807 14254 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Bootstrap starting.
I20260929 11:33:49.280776 14255 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Bootstrap starting.
I20260929 11:33:49.283064 14254 tablet_bootstrap.cc:654] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.284040 14254 log.cc:824] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:49.284148 14255 tablet_bootstrap.cc:654] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.285368 14255 log.cc:824] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:49.286739 14254 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: No bootstrap required, opened a new log
I20260929 11:33:49.287005 14254 ts_tablet_manager.cc:1397] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:33:49.287320 14258 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Bootstrap starting.
I20260929 11:33:49.287988 14255 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: No bootstrap required, opened a new log
I20260929 11:33:49.288234 14255 ts_tablet_manager.cc:1397] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:33:49.290683 14258 tablet_bootstrap.cc:654] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.292026 14258 log.cc:824] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:49.295951 14255 raft_consensus.cc:348] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.296360 14255 raft_consensus.cc:374] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.296479 14255 raft_consensus.cc:729] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.297266 14255 consensus_queue.cc:260] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.297679 14254 raft_consensus.cc:348] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.298167 14254 raft_consensus.cc:374] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.298322 14254 raft_consensus.cc:729] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.299052 14255 ts_tablet_manager.cc:1428] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 11:33:49.299431 14254 consensus_queue.cc:260] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.299739 14258 tablet_bootstrap.cc:492] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: No bootstrap required, opened a new log
I20260929 11:33:49.300086 14258 ts_tablet_manager.cc:1397] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260929 11:33:49.300541 14091 heartbeater.cc:503] Master 127.6.9.254:32947 was elected leader, sending a full tablet report...
I20260929 11:33:49.301200 14254 ts_tablet_manager.cc:1428] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 11:33:49.308866 14258 raft_consensus.cc:348] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.309257 14258 raft_consensus.cc:374] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.309358 14258 raft_consensus.cc:729] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.309911 14258 consensus_queue.cc:260] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.310897 14230 heartbeater.cc:503] Master 127.6.9.254:32947 was elected leader, sending a full tablet report...
I20260929 11:33:49.311425 14258 ts_tablet_manager.cc:1428] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260929 11:33:49.328418 14262 raft_consensus.cc:482] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:49.328747 14262 raft_consensus.cc:504] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.330413 14262 leader_election.cc:288] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551)
I20260929 11:33:49.336843 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.330827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:57235 (local address 127.6.9.193:43551)
0929 11:33:49.331095 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.331100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.331121 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:33:49.331351 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.331357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.331455 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.331559 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.332064 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.333059 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.334973 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.336010 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.336110 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.336185 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.336521 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.336616 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 11:33:49.336949 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.330832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43551 (local address 127.6.9.195:57235)
0929 11:33:49.331065 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.331084 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.331184 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.331601 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.331605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.331641 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.331875 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.331888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.333205 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.333211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.334792 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.334805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.336170 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.336176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.336361 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.336398 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.336491 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.336626 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1}
I20260929 11:33:49.337702 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.330820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:58183 (local address 127.6.9.194:36373)
0929 11:33:49.331026 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.331032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.331974 (+ 942us) server_negotiation.cc:407] Connection header received
0929 11:33:49.332108 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.332113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.332183 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.332255 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.332805 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.333807 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.335749 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.336788 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.336918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.337018 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.337349 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.337485 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:33:49.337715 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.330908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36373 (local address 127.6.9.195:58183)
0929 11:33:49.331864 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.331880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.332000 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.332290 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.332294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.332320 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.332639 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.332650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.333954 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.333958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.335586 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.335601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.337015 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.337020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.337181 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.337212 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.337305 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.337432 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":260,"thread_start_us":364,"threads_started":1}
I20260929 11:33:49.338165 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c1280afaaf4bfea3b1bcd1bbb84cba" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" is_pre_election: true
I20260929 11:33:49.338701 13906 raft_consensus.cc:2454] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 0.
I20260929 11:33:49.338649 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c1280afaaf4bfea3b1bcd1bbb84cba" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837" is_pre_election: true
I20260929 11:33:49.339087 14041 raft_consensus.cc:2454] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 0.
I20260929 11:33:49.339360 14113 leader_election.cc:302] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f378f76b47734b28a5d5b5e3c9424f10; no voters:
I20260929 11:33:49.339752 14262 raft_consensus.cc:2781] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:49.339924 14262 raft_consensus.cc:482] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:49.340023 14262 raft_consensus.cc:3037] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.342391 14262 raft_consensus.cc:504] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.342976 14262 leader_election.cc:288] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 election: Requested vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551)
I20260929 11:33:49.343326 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c1280afaaf4bfea3b1bcd1bbb84cba" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837"
I20260929 11:33:49.343437 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73c1280afaaf4bfea3b1bcd1bbb84cba" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
I20260929 11:33:49.343506 14041 raft_consensus.cc:3037] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.343636 13906 raft_consensus.cc:3037] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.345700 14041 raft_consensus.cc:2454] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 1.
I20260929 11:33:49.345744 13906 raft_consensus.cc:2454] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 1.
I20260929 11:33:49.346078 14112 leader_election.cc:302] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f378f76b47734b28a5d5b5e3c9424f10, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:49.346366 14262 raft_consensus.cc:2781] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:49.346813 14262 raft_consensus.cc:686] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 1 LEADER]: Becoming Leader. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Running, Role: LEADER
I20260929 11:33:49.347343 14262 consensus_queue.cc:237] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
W20260929 11:33:49.348958 13953 tablet.cc:2369] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:49.352134 13744 catalog_manager.cc:5495] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195). New cstate: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } health_report { overall_health: HEALTHY } } }
W20260929 11:33:49.369391 14092 tablet.cc:2369] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:49.376786 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.371617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59252 (local address 127.6.9.195:34747)
0929 11:33:49.371832 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.371838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.371885 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:33:49.372092 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.372096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.372185 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.372305 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:49.372570 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.373343 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.374578 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.374998 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.375089 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.375534 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.376132 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.376260 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.376397 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.376474 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:33:49.385254 13906 raft_consensus.cc:1261] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Refusing update from remote peer f378f76b47734b28a5d5b5e3c9424f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.385254 14041 raft_consensus.cc:1261] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Refusing update from remote peer f378f76b47734b28a5d5b5e3c9424f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.386291 14264 consensus_queue.cc:1035] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:49.386688 14262 consensus_queue.cc:1035] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:33:49.394399 14231 tablet.cc:2369] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:33:49.399107 14268 mvcc.cc:205] Tried to move back new op lower bound from 7334631953952174080 to 7334631953810579456. Current Snapshot: MvccSnapshot[applied={T|T < 7334631953952174080}]
I20260929 11:33:49.420804 13745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:49.421121 13745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:49.425549 13745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Sending DeleteTablet for 3 replicas of tablet 73c1280afaaf4bfea3b1bcd1bbb84cba
I20260929 11:33:49.427124 13882 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:49 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260929 11:33:49.427564 14160 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:49 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 11:33:49.427608 14021 tablet_service.cc:1514] Processing DeleteTablet for tablet 73c1280afaaf4bfea3b1bcd1bbb84cba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:49 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60740
I20260929 11:33:49.441141 13745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:49.442051 13745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:49.450762 14277 tablet_replica.cc:330] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: stopping tablet replica
I20260929 11:33:49.461359 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 5686d88cb1564a8b822f82803521248d (DEFAULT_TABLE table=test-table [id=2c12cb5d29194784a09e501e719369cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.461560 14279 tablet_replica.cc:330] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: stopping tablet replica
I20260929 11:33:49.462123 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5686d88cb1564a8b822f82803521248d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.462154 14279 raft_consensus.cc:2229] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:49.462698 14279 raft_consensus.cc:2258] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:49.465267 14159 tablet_service.cc:1467] Processing CreateTablet for tablet 5686d88cb1564a8b822f82803521248d (DEFAULT_TABLE table=test-table [id=2c12cb5d29194784a09e501e719369cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.465984 14159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5686d88cb1564a8b822f82803521248d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.467444 14279 ts_tablet_manager.cc:1905] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:49.473371 14258 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Bootstrap starting.
I20260929 11:33:49.475168 14254 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Bootstrap starting.
I20260929 11:33:49.475262 14258 tablet_bootstrap.cc:654] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.476081 14278 tablet_replica.cc:330] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: stopping tablet replica
I20260929 11:33:49.476838 14254 tablet_bootstrap.cc:654] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.476908 14278 raft_consensus.cc:2229] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:49.477656 14277 raft_consensus.cc:2229] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:49.460430 13881 tablet_service.cc:1467] Processing CreateTablet for tablet 5686d88cb1564a8b822f82803521248d (DEFAULT_TABLE table=test-table [id=2c12cb5d29194784a09e501e719369cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:49.478256 14277 raft_consensus.cc:2258] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:49.478363 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5686d88cb1564a8b822f82803521248d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.478932 14278 raft_consensus.cc:2258] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:49.481433 14277 ts_tablet_manager.cc:1905] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:49.483002 14278 ts_tablet_manager.cc:1905] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:49.487357 14254 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: No bootstrap required, opened a new log
I20260929 11:33:49.487623 14254 ts_tablet_manager.cc:1397] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20260929 11:33:49.488606 14254 raft_consensus.cc:348] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.489102 14254 raft_consensus.cc:374] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.489221 14277 ts_tablet_manager.cc:1918] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:49.489248 14254 raft_consensus.cc:729] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.489418 14277 log.cc:1196] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/73c1280afaaf4bfea3b1bcd1bbb84cba
I20260929 11:33:49.489735 14254 consensus_queue.cc:260] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.489950 14277 ts_tablet_manager.cc:1939] T 73c1280afaaf4bfea3b1bcd1bbb84cba P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting consensus metadata
I20260929 11:33:49.490345 14278 ts_tablet_manager.cc:1918] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:49.490522 14278 log.cc:1196] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/73c1280afaaf4bfea3b1bcd1bbb84cba
I20260929 11:33:49.491232 14254 ts_tablet_manager.cc:1428] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:33:49.491724 14255 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Bootstrap starting.
I20260929 11:33:49.493382 14278 ts_tablet_manager.cc:1939] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f378f76b47734b28a5d5b5e3c9424f10: Deleting consensus metadata
I20260929 11:33:49.493435 14255 tablet_bootstrap.cc:654] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.494570 13731 catalog_manager.cc:4841] TS 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551): tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (table test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]) successfully deleted
I20260929 11:33:49.495163 13730 catalog_manager.cc:4841] TS f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747): tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (table test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]) successfully deleted
I20260929 11:33:49.496598 14258 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: No bootstrap required, opened a new log
I20260929 11:33:49.496793 14258 ts_tablet_manager.cc:1397] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.005s
I20260929 11:33:49.497129 14279 ts_tablet_manager.cc:1918] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:49.497326 14279 log.cc:1196] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/73c1280afaaf4bfea3b1bcd1bbb84cba
I20260929 11:33:49.497699 14258 raft_consensus.cc:348] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.497888 14279 ts_tablet_manager.cc:1939] T 73c1280afaaf4bfea3b1bcd1bbb84cba P f41aafeadae0448c80e82cd5a05d7837: Deleting consensus metadata
I20260929 11:33:49.497962 14258 raft_consensus.cc:374] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.498065 14258 raft_consensus.cc:729] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.498926 14258 consensus_queue.cc:260] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.500053 13730 catalog_manager.cc:4841] TS f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373): tablet 73c1280afaaf4bfea3b1bcd1bbb84cba (table test-table [id=9f2e9ae6f06b42e3b4954f5f59dc0c91]) successfully deleted
I20260929 11:33:49.500202 14258 ts_tablet_manager.cc:1428] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:49.501142 14255 tablet_bootstrap.cc:492] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: No bootstrap required, opened a new log
I20260929 11:33:49.501310 14255 ts_tablet_manager.cc:1397] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 11:33:49.502202 14255 raft_consensus.cc:348] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.502454 14255 raft_consensus.cc:374] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.502553 14255 raft_consensus.cc:729] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.502846 14255 consensus_queue.cc:260] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.503516 14255 ts_tablet_manager.cc:1428] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:49.512429 14261 raft_consensus.cc:482] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:49.512732 14261 raft_consensus.cc:504] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.514742 14261 leader_election.cc:288] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:49.522655 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.515189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:37109 (local address 127.6.9.193:43551)
0929 11:33:49.515372 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.515379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.515972 (+ 593us) server_negotiation.cc:407] Connection header received
0929 11:33:49.516296 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.516302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.516379 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.516461 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.517619 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.518687 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.520510 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.521630 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.521757 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.521859 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.522237 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.522378 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:33:49.522951 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.514935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43551 (local address 127.6.9.194:37109)
0929 11:33:49.515858 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.515878 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.516121 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.516948 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.516954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.516986 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.517456 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.517469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.518818 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.518823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.520360 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.520372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.521837 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.521845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.522060 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.522102 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.522210 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.522353 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":42,"thread_start_us":624,"threads_started":1}
I20260929 11:33:49.523798 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5686d88cb1564a8b822f82803521248d" candidate_uuid: "f41aafeadae0448c80e82cd5a05d7837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" is_pre_election: true
I20260929 11:33:49.524075 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.515559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:45901 (local address 127.6.9.195:34747)
0929 11:33:49.515717 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.515722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.516268 (+ 546us) server_negotiation.cc:407] Connection header received
0929 11:33:49.516541 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.516548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.516630 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.516757 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:49.517307 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.518457 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.521239 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.522399 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.522518 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.522612 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.523604 (+ 992us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.523772 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:33:49.524075 13906 raft_consensus.cc:2454] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41aafeadae0448c80e82cd5a05d7837 in term 0.
I20260929 11:33:49.524799 13974 leader_election.cc:302] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:49.524964 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.515096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.6.9.194:45901)
0929 11:33:49.516175 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.516190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.516375 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.516774 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.516779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.516800 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:49.517141 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.517154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.518611 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.518617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.521047 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.521063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.523239 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.523246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.523424 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.523457 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.524307 (+ 850us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.524441 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":56,"thread_start_us":773,"threads_started":1}
I20260929 11:33:49.525305 14261 raft_consensus.cc:2781] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:49.525425 14261 raft_consensus.cc:482] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:49.525527 14261 raft_consensus.cc:3037] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.525873 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5686d88cb1564a8b822f82803521248d" candidate_uuid: "f41aafeadae0448c80e82cd5a05d7837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10" is_pre_election: true
I20260929 11:33:49.526417 14180 raft_consensus.cc:2454] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41aafeadae0448c80e82cd5a05d7837 in term 0.
I20260929 11:33:49.527973 14261 raft_consensus.cc:504] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.528564 14261 leader_election.cc:288] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [CANDIDATE]: Term 1 election: Requested vote from peers 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:49.529124 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5686d88cb1564a8b822f82803521248d" candidate_uuid: "f41aafeadae0448c80e82cd5a05d7837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10"
I20260929 11:33:49.529327 14180 raft_consensus.cc:3037] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.531633 14180 raft_consensus.cc:2454] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41aafeadae0448c80e82cd5a05d7837 in term 1.
I20260929 11:33:49.532120 13973 leader_election.cc:302] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f378f76b47734b28a5d5b5e3c9424f10, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:49.532112 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5686d88cb1564a8b822f82803521248d" candidate_uuid: "f41aafeadae0448c80e82cd5a05d7837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
I20260929 11:33:49.532336 13906 raft_consensus.cc:3037] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.532473 14261 raft_consensus.cc:2781] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:49.533007 14261 raft_consensus.cc:686] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 1 LEADER]: Becoming Leader. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Running, Role: LEADER
I20260929 11:33:49.533454 14261 consensus_queue.cc:237] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.535276 13906 raft_consensus.cc:2454] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41aafeadae0448c80e82cd5a05d7837 in term 1.
I20260929 11:33:49.538254 13745 catalog_manager.cc:5495] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 reported cstate change: term changed from 0 to 1, leader changed from <none> to f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "f41aafeadae0448c80e82cd5a05d7837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:33:49 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681627, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.254@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:49.593958 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.579160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:32947 (local address 127.0.0.1:53378)
0929 11:33:49.579852 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.579871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.580004 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.580337 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.580340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.581125 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:49.581443 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.581455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.582887 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.582891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.583879 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.583896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.584894 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.584898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.585011 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.585648 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:49.585665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:49.588009 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:49.591603 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.591616 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.591625 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.592039 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.592391 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:49.592394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:49.592396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:49.592531 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:49.593147 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:49.593154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:49.593310 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.593445 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.593641 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":381,"threads_started":1}
I20260929 11:33:49.594203 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.579157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.6.9.254:32947)
0929 11:33:49.579420 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.579427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.579959 (+ 532us) server_negotiation.cc:407] Connection header received
0929 11:33:49.580111 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.580117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.580199 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.580320 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:49.581579 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.582757 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.584031 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.584718 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.584853 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.588200 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:49.588207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:49.588212 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:49.588249 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:49.591473 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.592143 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.592149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.592153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.592296 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:49.592634 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:49.592639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:49.592642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:49.593032 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:49.593193 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.593579 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.593806 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 11:33:49.599020 13745 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:53378:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:33:49.600092 13745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:49.607267 14020 tablet_service.cc:1467] Processing CreateTablet for tablet e146d160ca374d6d9981ab559e9ef544 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34c76b6d0d06416e892418a72fc1ec1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:49.607267 13881 tablet_service.cc:1467] Processing CreateTablet for tablet e146d160ca374d6d9981ab559e9ef544 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34c76b6d0d06416e892418a72fc1ec1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:49.607823 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e146d160ca374d6d9981ab559e9ef544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.607818 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e146d160ca374d6d9981ab559e9ef544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.612795 14160 tablet_service.cc:1467] Processing CreateTablet for tablet e146d160ca374d6d9981ab559e9ef544 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34c76b6d0d06416e892418a72fc1ec1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:33:49.613343 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e146d160ca374d6d9981ab559e9ef544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:49.614866 14255 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: Bootstrap starting.
I20260929 11:33:49.614871 14254 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: Bootstrap starting.
I20260929 11:33:49.616482 14255 tablet_bootstrap.cc:654] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.616724 14254 tablet_bootstrap.cc:654] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.621416 14258 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: Bootstrap starting.
I20260929 11:33:49.622977 14258 tablet_bootstrap.cc:654] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:49.627000 14254 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: No bootstrap required, opened a new log
I20260929 11:33:49.627197 14254 ts_tablet_manager.cc:1397] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260929 11:33:49.628017 14254 raft_consensus.cc:348] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.628274 14254 raft_consensus.cc:374] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.628367 14254 raft_consensus.cc:729] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.628648 14254 consensus_queue.cc:260] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.629107 14261 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.629375 14261 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.629755 14254 ts_tablet_manager.cc:1428] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:33:49.631850 14255 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: No bootstrap required, opened a new log
I20260929 11:33:49.632074 14255 ts_tablet_manager.cc:1397] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260929 11:33:49.632951 14255 raft_consensus.cc:348] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.633270 14255 raft_consensus.cc:374] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.633361 14255 raft_consensus.cc:729] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.633606 14255 consensus_queue.cc:260] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.634298 14255 ts_tablet_manager.cc:1428] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 11:33:49.634141 14260 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.634510 14260 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.637725 14258 tablet_bootstrap.cc:492] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: No bootstrap required, opened a new log
I20260929 11:33:49.637905 14258 ts_tablet_manager.cc:1397] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260929 11:33:49.638541 14258 raft_consensus.cc:348] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.638739 14258 raft_consensus.cc:374] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:49.638827 14258 raft_consensus.cc:729] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Initialized, Role: FOLLOWER
I20260929 11:33:49.639063 14258 consensus_queue.cc:260] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.639624 14258 ts_tablet_manager.cc:1428] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 11:33:49.639495 14264 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.639828 14264 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.833993 14264 raft_consensus.cc:482] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:49.834280 14264 raft_consensus.cc:504] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.835158 14264 leader_election.cc:288] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551), f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373)
I20260929 11:33:49.835721 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e146d160ca374d6d9981ab559e9ef544" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" is_pre_election: true
I20260929 11:33:49.835776 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e146d160ca374d6d9981ab559e9ef544" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837" is_pre_election: true
I20260929 11:33:49.835973 13906 raft_consensus.cc:2454] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 0.
I20260929 11:33:49.836055 14041 raft_consensus.cc:2454] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 0.
I20260929 11:33:49.836416 14113 leader_election.cc:302] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f378f76b47734b28a5d5b5e3c9424f10; no voters:
I20260929 11:33:49.836755 14264 raft_consensus.cc:2781] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:49.836872 14264 raft_consensus.cc:482] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:49.836988 14264 raft_consensus.cc:3037] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.840121 14264 raft_consensus.cc:504] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.840865 14264 leader_election.cc:288] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 election: Requested vote from peers 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551), f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373)
I20260929 11:33:49.841377 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e146d160ca374d6d9981ab559e9ef544" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837"
I20260929 11:33:49.841387 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e146d160ca374d6d9981ab559e9ef544" candidate_uuid: "f378f76b47734b28a5d5b5e3c9424f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02"
I20260929 11:33:49.841585 14041 raft_consensus.cc:3037] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.841626 13906 raft_consensus.cc:3037] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:49.843607 14041 raft_consensus.cc:2454] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 1.
I20260929 11:33:49.844069 14112 leader_election.cc:302] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f378f76b47734b28a5d5b5e3c9424f10, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:49.844358 14264 raft_consensus.cc:2781] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:49.844539 14264 raft_consensus.cc:686] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 LEADER]: Becoming Leader. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Running, Role: LEADER
I20260929 11:33:49.844532 13906 raft_consensus.cc:2454] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f378f76b47734b28a5d5b5e3c9424f10 in term 1.
I20260929 11:33:49.844825 14264 consensus_queue.cc:237] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:49.846527 14275 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: TxnStatusTablet state changed. Reason: New leader f378f76b47734b28a5d5b5e3c9424f10. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.846858 14275 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:49.847522 14297 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:33:49.847741 14297 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:33:49.848147 13745 catalog_manager.cc:5495] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195). New cstate: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } health_report { overall_health: HEALTHY } } }
I20260929 11:33:49.877599 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.872716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.6.9.195:34747)
0929 11:33:49.872882 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.872891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.872966 (+ 75us) server_negotiation.cc:407] Connection header received
0929 11:33:49.873217 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.873222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.873327 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.873416 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:49.873764 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.874496 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.875742 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.876207 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.876330 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.876862 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.877032 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.877131 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.877295 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.877362 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:33:49.877812 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.872626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.0.0.1:59260)
0929 11:33:49.872883 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.872902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.873039 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.873449 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.873454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.873466 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:49.873640 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.873649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.874619 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.874625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.875609 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.875622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.876365 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.876371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.876521 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.876603 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.877154 (+ 551us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.877179 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.877271 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.877450 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:33:49.881295 14041 raft_consensus.cc:1261] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Refusing update from remote peer f378f76b47734b28a5d5b5e3c9424f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.881337 13906 raft_consensus.cc:1261] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Refusing update from remote peer f378f76b47734b28a5d5b5e3c9424f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.881927 14264 consensus_queue.cc:1035] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:49.882261 14275 consensus_queue.cc:1035] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:49.885744 14260 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: TxnStatusTablet state changed. Reason: New leader f378f76b47734b28a5d5b5e3c9424f10. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.886025 14260 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.887147 14261 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: TxnStatusTablet state changed. Reason: New leader f378f76b47734b28a5d5b5e3c9424f10. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.887446 14261 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.888113 14264 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.888422 14264 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:49.889932 14262 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.890177 14262 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f378f76b47734b28a5d5b5e3c9424f10: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:33:49.891135 14260 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.891373 14260 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P 89e0aa9cc8e5463598a5dc0bddef0b02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.892017 14261 tablet_replica.cc:439] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f378f76b47734b28a5d5b5e3c9424f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } } }
I20260929 11:33:49.892293 14261 tablet_replica.cc:442] T e146d160ca374d6d9981ab559e9ef544 P f41aafeadae0448c80e82cd5a05d7837: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:33:49.904146 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.897399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.6.9.194:36373)
0929 11:33:49.898696 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.898700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.898717 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:33:49.898784 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.898789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.898857 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.899005 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:49.899488 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.900626 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.901943 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.902368 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.902473 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.902923 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.903475 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.903604 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.903734 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.903833 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":1036,"threads_started":1}
I20260929 11:33:49.920751 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.915228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:42533 (local address 127.6.9.195:34747)
0929 11:33:49.915407 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.915414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.915903 (+ 489us) server_negotiation.cc:407] Connection header received
0929 11:33:49.916091 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.916096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.916166 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.916256 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:49.916763 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.917563 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.918994 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.919445 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.919563 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.919982 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.920096 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.920169 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.920314 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.920395 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:33:49.920809 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.915171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.6.9.194:42533)
0929 11:33:49.915788 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.915807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.915954 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.916280 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.916285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.916296 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:49.916598 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.916606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.917699 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.917704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.918854 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.918867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.919612 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.919619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.919747 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.919814 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.920194 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.920204 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.920270 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.920402 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":341,"threads_started":1}
I20260929 11:33:49.930300 13906 raft_consensus.cc:1261] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Refusing update from remote peer f41aafeadae0448c80e82cd5a05d7837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.930445 14180 raft_consensus.cc:1261] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Refusing update from remote peer f41aafeadae0448c80e82cd5a05d7837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:49.930958 14285 consensus_queue.cc:1035] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:49.931648 14261 consensus_queue.cc:1035] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:49.940500 14269 mvcc.cc:205] Tried to move back new op lower bound from 7334631956187111424 to 7334631954573848576. Current Snapshot: MvccSnapshot[applied={T|T < 7334631956187111424}]
I20260929 11:33:49.967028 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.960328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:47205 (local address 127.6.9.194:36373)
0929 11:33:49.960520 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:49.960526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:49.961272 (+ 746us) server_negotiation.cc:407] Connection header received
0929 11:33:49.961487 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:49.961493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:49.961578 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:49.961670 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:49.962482 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.963271 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.964920 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.965474 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.965600 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.966311 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.966456 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.966519 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:49.966723 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 11:33:49.966807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:33:49.967260 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:49.960299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36373 (local address 127.6.9.195:47205)
0929 11:33:49.961161 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:49.961180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:49.961337 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:49.961967 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:49.961971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:49.961985 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:49.962334 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.962344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.963405 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.963411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.964773 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:49.964797 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:49.965648 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:49.965655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:49.966033 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:49.966112 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:49.966595 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:49.966605 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:33:49.966709 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:33:49.966871 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":511,"threads_started":1}
I20260929 11:33:50.005414 13743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.005668 13743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.008747 13743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Sending DeleteTablet for 3 replicas of tablet 5686d88cb1564a8b822f82803521248d
I20260929 11:33:50.009648 13881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5686d88cb1564a8b822f82803521248d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260929 11:33:50.009874 14160 tablet_service.cc:1514] Processing DeleteTablet for tablet 5686d88cb1564a8b822f82803521248d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 11:33:50.010540 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5686d88cb1564a8b822f82803521248d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60740
I20260929 11:33:50.035458 14316 tablet_replica.cc:330] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: stopping tablet replica
I20260929 11:33:50.035712 14316 raft_consensus.cc:2229] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:50.036402 14316 raft_consensus.cc:2258] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.037513 14315 tablet_replica.cc:330] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: stopping tablet replica
I20260929 11:33:50.037730 14315 raft_consensus.cc:2229] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.038025 14315 raft_consensus.cc:2258] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.039487 14316 ts_tablet_manager.cc:1905] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.046005 14315 ts_tablet_manager.cc:1905] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.046838 14314 tablet_replica.cc:330] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: stopping tablet replica
I20260929 11:33:50.047025 14314 raft_consensus.cc:2229] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.047243 14314 raft_consensus.cc:2258] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.052651 14315 ts_tablet_manager.cc:1918] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.052858 14315 log.cc:1196] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/5686d88cb1564a8b822f82803521248d
I20260929 11:33:50.053324 14315 ts_tablet_manager.cc:1939] T 5686d88cb1564a8b822f82803521248d P f378f76b47734b28a5d5b5e3c9424f10: Deleting consensus metadata
I20260929 11:33:50.053504 14314 ts_tablet_manager.cc:1905] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.054605 13730 catalog_manager.cc:4841] TS f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747): tablet 5686d88cb1564a8b822f82803521248d (table test-table [id=2c12cb5d29194784a09e501e719369cc]) successfully deleted
I20260929 11:33:50.059890 14316 ts_tablet_manager.cc:1918] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.059887 14314 ts_tablet_manager.cc:1918] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.060055 14316 log.cc:1196] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/5686d88cb1564a8b822f82803521248d
I20260929 11:33:50.060207 14314 log.cc:1196] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/5686d88cb1564a8b822f82803521248d
I20260929 11:33:50.060485 14316 ts_tablet_manager.cc:1939] T 5686d88cb1564a8b822f82803521248d P f41aafeadae0448c80e82cd5a05d7837: Deleting consensus metadata
I20260929 11:33:50.060746 14314 ts_tablet_manager.cc:1939] T 5686d88cb1564a8b822f82803521248d P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting consensus metadata
I20260929 11:33:50.061741 13731 catalog_manager.cc:4841] TS 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551): tablet 5686d88cb1564a8b822f82803521248d (table test-table [id=2c12cb5d29194784a09e501e719369cc]) successfully deleted
I20260929 11:33:50.061741 13730 catalog_manager.cc:4841] TS f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373): tablet 5686d88cb1564a8b822f82803521248d (table test-table [id=2c12cb5d29194784a09e501e719369cc]) successfully deleted
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:33:50.105343 13840 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:50.085693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.6.9.254:32947)
0929 11:33:50.085915 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.085923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.086306 (+ 383us) server_negotiation.cc:407] Connection header received
0929 11:33:50.086501 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.086507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.086573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.086655 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:50.087934 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.088647 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.089973 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.090494 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.090609 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.105064 (+ 14455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53382: BlockingRecv error: recv got EOF from 127.0.0.1:53382 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM
I20260929 11:33:50.157011 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.138010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.6.9.254:32947)
0929 11:33:50.138177 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.138182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.138714 (+ 532us) server_negotiation.cc:407] Connection header received
0929 11:33:50.138893 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.138899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.138973 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.139059 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:50.140416 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.141519 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.143459 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.144147 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.144250 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.149733 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:50.149740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:50.149745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:50.149807 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:50.153481 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.154467 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.154474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.154478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.154648 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.155125 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.155130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.155133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.155513 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:50.155744 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.156352 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.156592 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:33:50.162232 13743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:50.163005 13743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:50.171648 13881 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4168abe674148a65f085c2f951bd5 (DEFAULT_TABLE table=test-table [id=c627991a52334b0e87712f9091ebbf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.172178 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4168abe674148a65f085c2f951bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.172153 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4168abe674148a65f085c2f951bd5 (DEFAULT_TABLE table=test-table [id=c627991a52334b0e87712f9091ebbf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.172581 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4168abe674148a65f085c2f951bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.176795 14160 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4168abe674148a65f085c2f951bd5 (DEFAULT_TABLE table=test-table [id=c627991a52334b0e87712f9091ebbf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.177266 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4168abe674148a65f085c2f951bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.191331 14329 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Bootstrap starting.
I20260929 11:33:50.193085 14329 tablet_bootstrap.cc:654] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.207773 14332 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Bootstrap starting.
I20260929 11:33:50.209052 14331 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Bootstrap starting.
I20260929 11:33:50.213598 14331 tablet_bootstrap.cc:654] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.215018 14332 tablet_bootstrap.cc:654] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.215126 14329 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: No bootstrap required, opened a new log
I20260929 11:33:50.215303 14329 ts_tablet_manager.cc:1397] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20260929 11:33:50.216109 14329 raft_consensus.cc:348] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.216391 14329 raft_consensus.cc:374] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.216483 14329 raft_consensus.cc:729] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.216758 14329 consensus_queue.cc:260] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.217450 14329 ts_tablet_manager.cc:1428] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 11:33:50.228816 14332 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: No bootstrap required, opened a new log
I20260929 11:33:50.229100 14332 ts_tablet_manager.cc:1397] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20260929 11:33:50.230142 14332 raft_consensus.cc:348] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.230499 14332 raft_consensus.cc:374] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.230644 14332 raft_consensus.cc:729] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.231078 14332 consensus_queue.cc:260] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.231760 14332 ts_tablet_manager.cc:1428] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:50.239643 14331 tablet_bootstrap.cc:492] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: No bootstrap required, opened a new log
I20260929 11:33:50.239979 14331 ts_tablet_manager.cc:1397] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.005s
I20260929 11:33:50.241055 14331 raft_consensus.cc:348] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.241458 14331 raft_consensus.cc:374] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.241622 14331 raft_consensus.cc:729] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.241940 14331 consensus_queue.cc:260] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.242616 14331 ts_tablet_manager.cc:1428] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:50.270444 14260 raft_consensus.cc:482] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:50.270663 14260 raft_consensus.cc:504] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.272223 14260 leader_election.cc:288] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:50.281705 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.272637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:33353 (local address 127.6.9.194:36373)
0929 11:33:50.272943 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.272954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.273291 (+ 337us) server_negotiation.cc:407] Connection header received
0929 11:33:50.273629 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.273635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.273713 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.273785 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:50.274627 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.275911 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.278853 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.280287 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.280398 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.280524 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.281147 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.281332 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 11:33:50.282855 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.272537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36373 (local address 127.6.9.193:33353)
0929 11:33:50.273189 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:50.273206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:50.273408 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:50.274008 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:50.274013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:50.274042 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:50.274413 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.274426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.276115 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.276121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.278623 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.278644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.280716 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.280722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.280933 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.280978 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 11:33:50.282261 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0929 11:33:50.282446 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":422,"threads_started":1}
I20260929 11:33:50.285984 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4168abe674148a65f085c2f951bd5" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837" is_pre_election: true
I20260929 11:33:50.286314 14041 raft_consensus.cc:2454] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 0.
I20260929 11:33:50.288121 13833 leader_election.cc:302] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:50.288601 14260 raft_consensus.cc:2781] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:50.288750 14260 raft_consensus.cc:482] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:50.288847 14260 raft_consensus.cc:3037] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.291519 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.273752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:42917 (local address 127.6.9.195:34747)
0929 11:33:50.273938 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.273943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.283785 (+ 9842us) server_negotiation.cc:407] Connection header received
0929 11:33:50.284091 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.284097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.284181 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.284249 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:33:50.284902 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.286002 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.288730 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.290002 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.290121 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.290249 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.291063 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.291232 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:33:50.292167 14260 raft_consensus.cc:504] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.292687 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.273388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.6.9.193:42917)
0929 11:33:50.283671 (+ 10283us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:50.283694 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:50.283926 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:50.284471 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:50.284476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:50.284504 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:33:50.284723 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.284735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.286651 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.286657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.288499 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.288516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.290538 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.290543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.290743 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.290776 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:33:50.292113 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0929 11:33:50.292305 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10161,"thread_start_us":11770,"threads_started":1}
I20260929 11:33:50.293033 14260 leader_election.cc:288] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 election: Requested vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:50.293815 14180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4168abe674148a65f085c2f951bd5" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10" is_pre_election: true
I20260929 11:33:50.294095 14180 raft_consensus.cc:2454] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 0.
I20260929 11:33:50.294173 14179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4168abe674148a65f085c2f951bd5" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10"
I20260929 11:33:50.294427 14179 raft_consensus.cc:3037] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.295224 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4168abe674148a65f085c2f951bd5" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837"
I20260929 11:33:50.295459 14041 raft_consensus.cc:3037] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.297819 14179 raft_consensus.cc:2454] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 1.
I20260929 11:33:50.298339 13833 leader_election.cc:302] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f378f76b47734b28a5d5b5e3c9424f10; no voters:
I20260929 11:33:50.298748 14260 raft_consensus.cc:2781] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:50.298738 14041 raft_consensus.cc:2454] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 1.
I20260929 11:33:50.299623 14260 raft_consensus.cc:686] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 LEADER]: Becoming Leader. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Running, Role: LEADER
I20260929 11:33:50.300110 14260 consensus_queue.cc:237] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.303634 13743 catalog_manager.cc:5495] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:50.326506 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.316788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.6.9.193:43551)
0929 11:33:50.321117 (+ 4329us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.321122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.321143 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:33:50.321236 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.321240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.321321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.321484 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:50.321936 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.322749 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.324123 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.324553 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.324698 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.325156 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:50.325749 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:50.325903 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.326003 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.326089 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4215,"thread_start_us":4081,"threads_started":1}
I20260929 11:33:50.330157 14179 raft_consensus.cc:1261] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Refusing update from remote peer 89e0aa9cc8e5463598a5dc0bddef0b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:50.330823 14041 raft_consensus.cc:1261] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Refusing update from remote peer 89e0aa9cc8e5463598a5dc0bddef0b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:50.331192 14337 consensus_queue.cc:1035] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:50.331934 14260 consensus_queue.cc:1035] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:50.370890 13743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.371145 13743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.375085 13743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Sending DeleteTablet for 3 replicas of tablet 9aa4168abe674148a65f085c2f951bd5
I20260929 11:33:50.375927 13881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4168abe674148a65f085c2f951bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260929 11:33:50.376257 14314 tablet_replica.cc:330] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: stopping tablet replica
I20260929 11:33:50.376504 14314 raft_consensus.cc:2229] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:50.377722 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4168abe674148a65f085c2f951bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60740
I20260929 11:33:50.378050 14316 tablet_replica.cc:330] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: stopping tablet replica
I20260929 11:33:50.378259 14316 raft_consensus.cc:2229] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.378505 14316 raft_consensus.cc:2258] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.378666 14160 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4168abe674148a65f085c2f951bd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 11:33:50.379163 14315 tablet_replica.cc:330] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: stopping tablet replica
I20260929 11:33:50.379388 14315 raft_consensus.cc:2229] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.379652 14315 raft_consensus.cc:2258] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.381062 14316 ts_tablet_manager.cc:1905] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.386662 13744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:33:50.387406 13744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:33:50.397270 13881 tablet_service.cc:1467] Processing CreateTablet for tablet 7de88a7239aa498c971d025a3e307130 (DEFAULT_TABLE table=test-table [id=5fc293299d014225b49860abc090efc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.398774 14020 tablet_service.cc:1467] Processing CreateTablet for tablet 7de88a7239aa498c971d025a3e307130 (DEFAULT_TABLE table=test-table [id=5fc293299d014225b49860abc090efc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.399278 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de88a7239aa498c971d025a3e307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.401063 14160 tablet_service.cc:1467] Processing CreateTablet for tablet 7de88a7239aa498c971d025a3e307130 (DEFAULT_TABLE table=test-table [id=5fc293299d014225b49860abc090efc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:33:50.401561 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de88a7239aa498c971d025a3e307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.405061 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de88a7239aa498c971d025a3e307130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:50.428078 14332 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Bootstrap starting.
I20260929 11:33:50.429996 14332 tablet_bootstrap.cc:654] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.435309 14316 ts_tablet_manager.cc:1918] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:50.435474 14316 log.cc:1196] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/9aa4168abe674148a65f085c2f951bd5
I20260929 11:33:50.437878 14329 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Bootstrap starting.
I20260929 11:33:50.439745 14329 tablet_bootstrap.cc:654] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.439765 14331 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Bootstrap starting.
I20260929 11:33:50.440058 14316 ts_tablet_manager.cc:1939] T 9aa4168abe674148a65f085c2f951bd5 P f41aafeadae0448c80e82cd5a05d7837: Deleting consensus metadata
I20260929 11:33:50.441987 14315 ts_tablet_manager.cc:1905] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.442441 14331 tablet_bootstrap.cc:654] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:50.450934 14315 ts_tablet_manager.cc:1918] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:50.451133 14315 log.cc:1196] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/9aa4168abe674148a65f085c2f951bd5
I20260929 11:33:50.451582 14315 ts_tablet_manager.cc:1939] T 9aa4168abe674148a65f085c2f951bd5 P f378f76b47734b28a5d5b5e3c9424f10: Deleting consensus metadata
I20260929 11:33:50.451993 13730 catalog_manager.cc:4841] TS f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373): tablet 9aa4168abe674148a65f085c2f951bd5 (table test-table [id=c627991a52334b0e87712f9091ebbf68]) successfully deleted
I20260929 11:33:50.452332 14331 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: No bootstrap required, opened a new log
I20260929 11:33:50.452507 14331 ts_tablet_manager.cc:1397] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260929 11:33:50.453135 13730 catalog_manager.cc:4841] TS f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747): tablet 9aa4168abe674148a65f085c2f951bd5 (table test-table [id=c627991a52334b0e87712f9091ebbf68]) successfully deleted
I20260929 11:33:50.453346 14331 raft_consensus.cc:348] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.453593 14331 raft_consensus.cc:374] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.453661 14331 raft_consensus.cc:729] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41aafeadae0448c80e82cd5a05d7837, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.453722 14314 raft_consensus.cc:2258] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.453931 14331 consensus_queue.cc:260] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.454962 14314 ts_tablet_manager.cc:1905] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.457006 14331 ts_tablet_manager.cc:1428] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 11:33:50.464422 14314 ts_tablet_manager.cc:1918] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:33:50.464643 14314 log.cc:1196] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/9aa4168abe674148a65f085c2f951bd5
I20260929 11:33:50.467943 14314 ts_tablet_manager.cc:1939] T 9aa4168abe674148a65f085c2f951bd5 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting consensus metadata
I20260929 11:33:50.468035 14332 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: No bootstrap required, opened a new log
I20260929 11:33:50.468232 14332 ts_tablet_manager.cc:1397] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.005s
I20260929 11:33:50.469309 13731 catalog_manager.cc:4841] TS 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551): tablet 9aa4168abe674148a65f085c2f951bd5 (table test-table [id=c627991a52334b0e87712f9091ebbf68]) successfully deleted
I20260929 11:33:50.469436 14329 tablet_bootstrap.cc:492] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: No bootstrap required, opened a new log
I20260929 11:33:50.469617 14329 ts_tablet_manager.cc:1397] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.001s
I20260929 11:33:50.469636 14332 raft_consensus.cc:348] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.469918 14332 raft_consensus.cc:374] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.470001 14332 raft_consensus.cc:729] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378f76b47734b28a5d5b5e3c9424f10, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.470283 14332 consensus_queue.cc:260] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.470423 14329 raft_consensus.cc:348] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.470700 14329 raft_consensus.cc:374] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:50.470798 14329 raft_consensus.cc:729] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Initialized, Role: FOLLOWER
I20260929 11:33:50.471072 14329 consensus_queue.cc:260] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.471593 14329 ts_tablet_manager.cc:1428] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:33:50.477041 14332 ts_tablet_manager.cc:1428] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260929 11:33:50.485284 14337 raft_consensus.cc:482] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:33:50.485524 14337 raft_consensus.cc:504] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.486261 14337 leader_election.cc:288] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:50.487349 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de88a7239aa498c971d025a3e307130" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837" is_pre_election: true
I20260929 11:33:50.487360 14179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de88a7239aa498c971d025a3e307130" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10" is_pre_election: true
I20260929 11:33:50.487598 14041 raft_consensus.cc:2454] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 0.
I20260929 11:33:50.487608 14179 raft_consensus.cc:2454] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 0.
I20260929 11:33:50.488096 13833 leader_election.cc:302] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:50.488606 14337 raft_consensus.cc:2781] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:33:50.488750 14337 raft_consensus.cc:482] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:33:50.488864 14337 raft_consensus.cc:3037] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.492002 14337 raft_consensus.cc:504] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.493135 14041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de88a7239aa498c971d025a3e307130" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41aafeadae0448c80e82cd5a05d7837"
I20260929 11:33:50.493362 14041 raft_consensus.cc:3037] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.494310 14337 leader_election.cc:288] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 election: Requested vote from peers f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373), f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747)
I20260929 11:33:50.494638 14179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de88a7239aa498c971d025a3e307130" candidate_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f378f76b47734b28a5d5b5e3c9424f10"
I20260929 11:33:50.494863 14179 raft_consensus.cc:3037] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:50.496330 14041 raft_consensus.cc:2454] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 1.
I20260929 11:33:50.496811 13833 leader_election.cc:302] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e0aa9cc8e5463598a5dc0bddef0b02, f41aafeadae0448c80e82cd5a05d7837; no voters:
I20260929 11:33:50.497144 14337 raft_consensus.cc:2781] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:50.497552 14179 raft_consensus.cc:2454] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e0aa9cc8e5463598a5dc0bddef0b02 in term 1.
I20260929 11:33:50.497592 14337 raft_consensus.cc:686] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 LEADER]: Becoming Leader. State: Replica: 89e0aa9cc8e5463598a5dc0bddef0b02, State: Running, Role: LEADER
I20260929 11:33:50.498085 14337 consensus_queue.cc:237] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } }
I20260929 11:33:50.512523 13743 catalog_manager.cc:5495] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89e0aa9cc8e5463598a5dc0bddef0b02" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 } health_report { overall_health: UNKNOWN } } }
I20260929 11:33:50.583133 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.574709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:47043 (local address 127.6.9.195:34747)
0929 11:33:50.574929 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.574937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.575678 (+ 741us) server_negotiation.cc:407] Connection header received
0929 11:33:50.575958 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.575964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.576069 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.576166 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:50.577011 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.578164 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.580598 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.581322 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.581460 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.582211 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:50.582407 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:50.582505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.582706 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.582810 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 11:33:50.583267 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.574739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:34747 (local address 127.6.9.193:47043)
0929 11:33:50.575533 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:50.575555 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:50.575760 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:50.576224 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:50.576230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:50.576244 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:50.576753 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.576771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.578400 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.578406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.580382 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.580405 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.581644 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.581652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.581871 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.581962 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:50.582545 (+ 583us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:50.582557 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 11:33:50.582658 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:33:50.582826 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":406,"threads_started":1}
I20260929 11:33:50.595062 14179 raft_consensus.cc:1261] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Refusing update from remote peer 89e0aa9cc8e5463598a5dc0bddef0b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:50.595070 14041 raft_consensus.cc:1261] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Refusing update from remote peer 89e0aa9cc8e5463598a5dc0bddef0b02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:33:50.596098 14337 consensus_queue.cc:1035] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f378f76b47734b28a5d5b5e3c9424f10" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:50.596555 14344 consensus_queue.cc:1035] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41aafeadae0448c80e82cd5a05d7837" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:33:50.671809 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:48929 (local address 127.6.9.193:43551)
0929 11:33:50.642859 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.642866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.656181 (+ 13315us) server_negotiation.cc:407] Connection header received
0929 11:33:50.656709 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.656714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.656813 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.656882 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:33:50.657578 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.658724 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.666689 (+ 7965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.667448 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.667554 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.670957 (+ 3403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:33:50.671106 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:33:50.671163 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.671364 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.671463 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752}
I20260929 11:33:50.672770 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.641580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43551 (local address 127.6.9.195:48929)
0929 11:33:50.656058 (+ 14478us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:50.656078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:50.656505 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:50.657135 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:50.657140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:50.657153 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:33:50.657422 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.657434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.664779 (+ 7345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.664785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.666466 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:50.666489 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.670442 (+ 3953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.670449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:50.670667 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.670730 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:33:50.671241 (+ 511us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:33:50.671251 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:33:50.672207 (+ 956us) client_negotiation.cc:240] Negotiation successful
0929 11:33:50.672378 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14295,"thread_start_us":14643,"threads_started":1}
I20260929 11:33:50.718007 13743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.718230 13743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:33:50.721207 13743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6b95a4c9674b13a045e105a740585c: Sending DeleteTablet for 3 replicas of tablet 7de88a7239aa498c971d025a3e307130
I20260929 11:33:50.722152 14020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de88a7239aa498c971d025a3e307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:60740
I20260929 11:33:50.722139 13881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de88a7239aa498c971d025a3e307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260929 11:33:50.722472 14314 tablet_replica.cc:330] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: stopping tablet replica
I20260929 11:33:50.722654 14314 raft_consensus.cc:2229] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:33:50.723019 14314 raft_consensus.cc:2258] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.723132 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13816
I20260929 11:33:50.723353 14316 tablet_replica.cc:330] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: stopping tablet replica
I20260929 11:33:50.723531 14316 raft_consensus.cc:2229] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.723732 14316 raft_consensus.cc:2258] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.724962 14316 ts_tablet_manager.cc:1905] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.729516 14314 ts_tablet_manager.cc:1905] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.732347 14160 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de88a7239aa498c971d025a3e307130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:33:50 UTC) from {username='oryx', principal='oryx/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 11:33:50.732738 14315 tablet_replica.cc:330] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: stopping tablet replica
I20260929 11:33:50.732954 14315 raft_consensus.cc:2229] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:33:50.733172 14315 raft_consensus.cc:2258] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:33:50.734678 14315 ts_tablet_manager.cc:1905] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:33:50.735831 14316 ts_tablet_manager.cc:1918] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.735844 14314 ts_tablet_manager.cc:1918] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.735978 14314 log.cc:1196] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/7de88a7239aa498c971d025a3e307130
I20260929 11:33:50.736121 14316 log.cc:1196] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/7de88a7239aa498c971d025a3e307130
I20260929 11:33:50.736413 14314 ts_tablet_manager.cc:1939] T 7de88a7239aa498c971d025a3e307130 P 89e0aa9cc8e5463598a5dc0bddef0b02: Deleting consensus metadata
I20260929 11:33:50.736600 14316 ts_tablet_manager.cc:1939] T 7de88a7239aa498c971d025a3e307130 P f41aafeadae0448c80e82cd5a05d7837: Deleting consensus metadata
I20260929 11:33:50.737742 13731 catalog_manager.cc:4841] TS 89e0aa9cc8e5463598a5dc0bddef0b02 (127.6.9.193:43551): tablet 7de88a7239aa498c971d025a3e307130 (table test-table [id=5fc293299d014225b49860abc090efc4]) successfully deleted
I20260929 11:33:50.737744 13730 catalog_manager.cc:4841] TS f41aafeadae0448c80e82cd5a05d7837 (127.6.9.194:36373): tablet 7de88a7239aa498c971d025a3e307130 (table test-table [id=5fc293299d014225b49860abc090efc4]) successfully deleted
I20260929 11:33:50.741360 14315 ts_tablet_manager.cc:1918] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:33:50.741499 14315 log.cc:1196] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/7de88a7239aa498c971d025a3e307130
I20260929 11:33:50.741937 14315 ts_tablet_manager.cc:1939] T 7de88a7239aa498c971d025a3e307130 P f378f76b47734b28a5d5b5e3c9424f10: Deleting consensus metadata
I20260929 11:33:50.743100 13730 catalog_manager.cc:4841] TS f378f76b47734b28a5d5b5e3c9424f10 (127.6.9.195:34747): tablet 7de88a7239aa498c971d025a3e307130 (table test-table [id=5fc293299d014225b49860abc090efc4]) successfully deleted
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.193@KRBTEST.COM
I20260929 11:33:50.753957 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.724484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.6.9.193:43551)
0929 11:33:50.728866 (+ 4382us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.728871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.738459 (+ 9588us) server_negotiation.cc:407] Connection header received
0929 11:33:50.738698 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.738702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.738768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.738869 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:50.740084 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.740925 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.742236 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.742673 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.742787 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.746818 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:50.746832 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:50.746847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:50.746898 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:50.750125 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.750888 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.750896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.750899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.751045 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.751446 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.751451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.751453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.751853 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:50.752196 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.753292 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.753552 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4217}
I20260929 11:33:50.754879 13947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:50.883982 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13816
I20260929 11:33:50.901247 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13956
Sep 29 11:33:50 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.194@KRBTEST.COM
I20260929 11:33:50.917171 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:50.902059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.6.9.194:36373)
0929 11:33:50.902598 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:50.902603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:50.902618 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:33:50.902671 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:50.902674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:50.902736 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:50.902840 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:50.904068 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.904896 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.906417 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:50.907023 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:50.907182 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:50.911578 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:50.911595 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:50.911612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:50.911656 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:50.914597 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.915334 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.915342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.915345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.915535 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:50.915878 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:50.915881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:50.915883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:50.916227 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:50.916446 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:50.916719 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 11:33:50.916912 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":327,"threads_started":1}
I20260929 11:33:50.917852 14086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:51.041695 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13956
I20260929 11:33:51.059477 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14095
Sep 29 11:33:51 dist-test-slave-c0l9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.9.195@KRBTEST.COM
I20260929 11:33:51.074954 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:51.060351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.6.9.195:34747)
0929 11:33:51.060563 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:51.060568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:51.060586 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:51.060642 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:51.060646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:51.060773 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:51.060883 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:51.062152 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:51.062872 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:51.064299 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:51.064881 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:51.065004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:51.069223 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:51.069240 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:51.069258 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:51.069315 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:51.072092 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:51.073016 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:51.073024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:51.073027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:51.073169 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:51.073532 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:51.073535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:51.073537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:51.073903 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:51.074148 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:51.074467 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 11:33:51.074657 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:33:51.075703 14225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:51.223716 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:51.222648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:36373 (local address 127.6.9.195:40597)
0929 11:33:51.223412 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:51.223548 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.194:36373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":568,"threads_started":1}
W20260929 11:33:51.224112 14112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.194:36373: connect: Connection refused (error 111)
I20260929 11:33:51.224784 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14095
I20260929 11:33:51.244953 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 13713
I20260929 11:33:51.246197 13785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:51.362015 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 13713
2026-09-29T11:33:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5579 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:51 dist-test-slave-c0l9 krb5kdc[14371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:51 dist-test-slave-c0l9 krb5kdc[14371](info): set up 2 sockets
Sep 29 11:33:51 dist-test-slave-c0l9 krb5kdc[14371](info): commencing operation
krb5kdc: starting...
W20260929 11:33:52.389393 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:52 dist-test-slave-c0l9 krb5kdc[14371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:52.597946 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:34207
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:37865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:34207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:52.927848 14384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:52.928143 14384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:52.928257 14384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:52.935554 14384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:52.935678 14384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:52.935729 14384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:52.935813 14384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:52.935876 14384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:52.946097 14384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:34207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:34207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:52.947796 14384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:52.949337 14384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:52.965003 14389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:52.964965 14392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:52.964933 14390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:52.965950 14384 server_base.cc:1034] running on GCE node
I20260929 11:33:52.966811 14384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:52.968482 14384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:52.969692 14384 hybrid_clock.cc:648] HybridClock initialized: now 1790681632969637 us; error 53 us; skew 500 ppm
I20260929 11:33:52.970042 14384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T11:33:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1676 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:33:53 dist-test-slave-c0l9 krb5kdc[14398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:33:53 dist-test-slave-c0l9 krb5kdc[14398](info): set up 2 sockets
Sep 29 11:33:53 dist-test-slave-c0l9 krb5kdc[14398](info): commencing operation
krb5kdc: starting...
W20260929 11:33:54.057250 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:33:54 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:33:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:33:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:33:54.272704 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36939
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:36939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:54.596912 14411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:54.597216 14411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:54.597349 14411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:54.604704 14411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:54.604823 14411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:54.604882 14411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:54.604960 14411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:54.605022 14411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:54.615224 14411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:36939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:54.616510 14411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:54.618101 14411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:54.629249 14416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:54.629215 14417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:54.629278 14419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:54.630098 14411 server_base.cc:1034] running on GCE node
I20260929 11:33:54.630883 14411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:54.632341 14411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:54.633561 14411 hybrid_clock.cc:648] HybridClock initialized: now 1790681634633509 us; error 39 us; skew 500 ppm
I20260929 11:33:54.633884 14411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:54 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:54.637012 14411 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:54.638439 14411 webserver.cc:460] Webserver started at http://127.6.9.254:35691/ using document root <none> and password file <none>
I20260929 11:33:54.639113 14411 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:54.639217 14411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:54.639523 14411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:54.642517 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "e0ad4574c5a84da6ab52f47258764b9f"
format_stamp: "Formatted at 2026-09-29 11:33:54 on dist-test-slave-c0l9"
server_key: "c168526690fe9802f25baba96a095db6"
server_key_iv: "b5fc47b6bf50bcd9f257ec5f87b30a56"
server_key_version: "encryptionkey@0"
I20260929 11:33:54.643312 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "e0ad4574c5a84da6ab52f47258764b9f"
format_stamp: "Formatted at 2026-09-29 11:33:54 on dist-test-slave-c0l9"
server_key: "c168526690fe9802f25baba96a095db6"
server_key_iv: "b5fc47b6bf50bcd9f257ec5f87b30a56"
server_key_version: "encryptionkey@0"
I20260929 11:33:54.648391 14411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:33:54.652073 14426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:54.653602 14411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:54.653785 14411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "e0ad4574c5a84da6ab52f47258764b9f"
format_stamp: "Formatted at 2026-09-29 11:33:54 on dist-test-slave-c0l9"
server_key: "c168526690fe9802f25baba96a095db6"
server_key_iv: "b5fc47b6bf50bcd9f257ec5f87b30a56"
server_key_version: "encryptionkey@0"
I20260929 11:33:54.653975 14411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:54.665555 14411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:54.669544 14411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:54.669798 14411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:54.686852 14411 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:36939
I20260929 11:33:54.686882 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:36939 every 8 connection(s)
I20260929 11:33:54.688329 14411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:54.692106 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:33:54.693495 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14411
I20260929 11:33:54.693696 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:33:54.694489 6183 external_mini_cluster.cc:1428] Setting key eb42784cbad4b228d87181834023779c
I20260929 11:33:54.706677 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Bootstrap starting.
Sep 29 11:33:54 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:54.710394 14478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Neither blocks nor log segments found. Creating new log.
I20260929 11:33:54.711696 14478 log.cc:824] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:54.714946 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: No bootstrap required, opened a new log
I20260929 11:33:54.718181 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:54.696620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.6.9.254:36939)
0929 11:33:54.697470 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:54.697485 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:54.697530 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:33:54.699111 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:54.699135 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:54.699568 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:54.699927 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:54.701641 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:54.702965 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:54.704787 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:54.705482 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:54.705616 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:54.710092 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:54.710102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:54.710122 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:54.710169 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:54.713749 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:54.714432 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:54.714440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:54.714447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:54.714642 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:54.715015 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:54.715020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:54.715023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:54.715694 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:54.715992 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:54.716319 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 11:33:54.716782 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":365,"threads_started":1}
I20260929 11:33:54.724311 14478 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:54.724778 14478 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:33:54.724889 14478 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ad4574c5a84da6ab52f47258764b9f, State: Initialized, Role: FOLLOWER
I20260929 11:33:54.725531 14478 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:54.725800 14478 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:54.725957 14478 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:54.726105 14478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:33:54.728363 14478 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:54.728968 14478 leader_election.cc:302] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ad4574c5a84da6ab52f47258764b9f; no voters:
I20260929 11:33:54.729607 14478 leader_election.cc:288] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:33:54.729707 14483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:33:54.730342 14483 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 LEADER]: Becoming Leader. State: Replica: e0ad4574c5a84da6ab52f47258764b9f, State: Running, Role: LEADER
I20260929 11:33:54.730867 14483 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:54.731622 14478 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:54.733749 14485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ad4574c5a84da6ab52f47258764b9f. Latest consensus state: current_term: 1 leader_uuid: "e0ad4574c5a84da6ab52f47258764b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } } }
I20260929 11:33:54.734201 14485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:54.734208 14484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ad4574c5a84da6ab52f47258764b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } } }
I20260929 11:33:54.734869 14484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:54.736268 14492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:54.740835 14492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:54.749886 14492 catalog_manager.cc:1347] Generated new cluster ID: 5c9d09da6b764301a8ffe7d32d1329d1
I20260929 11:33:54.750025 14492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:54.777648 14492 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:33:54.779574 14492 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:54.791504 14492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Generated new TSK 0
I20260929 11:33:54.792588 14492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:33:54.851845 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:55.181800 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:55.182117 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:55.182260 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:55.189666 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:55.189811 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:55.189975 14504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:55.200028 14504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:55.201373 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:55.202673 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:55.215373 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:55.215374 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:55.215350 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:55.216303 14504 server_base.cc:1034] running on GCE node
I20260929 11:33:55.216960 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:55.217973 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:55.219193 14504 hybrid_clock.cc:648] HybridClock initialized: now 1790681635219128 us; error 62 us; skew 500 ppm
I20260929 11:33:55.219525 14504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:55 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:55.222772 14504 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:33:55.224272 14504 webserver.cc:460] Webserver started at http://127.6.9.193:40315/ using document root <none> and password file <none>
I20260929 11:33:55.225003 14504 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:55.225121 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:55.225414 14504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:55.228317 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "65292315cdf34a2eaca8e733b1d32de7"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "8dc22db4f2a4bd4844e4296735c23590"
server_key_iv: "c89192ced9ec5e32debbf1f30334e6f0"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.229115 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "65292315cdf34a2eaca8e733b1d32de7"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "8dc22db4f2a4bd4844e4296735c23590"
server_key_iv: "c89192ced9ec5e32debbf1f30334e6f0"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.234480 14504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:33:55.238101 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.239404 14504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:33:55.239555 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "65292315cdf34a2eaca8e733b1d32de7"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "8dc22db4f2a4bd4844e4296735c23590"
server_key_iv: "c89192ced9ec5e32debbf1f30334e6f0"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.239748 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:55.262573 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:55.266636 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:55.266912 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:55.267894 14504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:55.269248 14504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:55.269332 14504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.269450 14504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:55.269503 14504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.300148 14504 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:44453
I20260929 11:33:55.300199 14631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:44453 every 8 connection(s)
I20260929 11:33:55.301409 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:55.305279 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14504
I20260929 11:33:55.305477 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:33:55.306103 6183 external_mini_cluster.cc:1428] Setting key a7e8079ed88e97626ece034d1fe81fba
Sep 29 11:33:55 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:55.320611 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:55.303911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:43965 (local address 127.6.9.254:36939)
0929 11:33:55.304377 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:55.304383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:55.305370 (+ 987us) server_negotiation.cc:407] Connection header received
0929 11:33:55.307327 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:55.307333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:55.307430 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:55.307523 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:55.309047 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.309761 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.311275 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.311667 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.311786 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:55.315152 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:55.315157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:55.315161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:55.315194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:55.317637 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:55.318346 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:55.318350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:55.318352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:55.318465 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:55.318866 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:55.318869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:55.318872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:55.319138 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:55.319267 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:55.320027 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 11:33:55.320258 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":201,"threads_started":1}
I20260929 11:33:55.321650 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:55.304456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:43965)
0929 11:33:55.305194 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:55.305238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:55.307076 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:55.307767 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:55.307775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:55.308277 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:55.308894 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:55.308912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.309908 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.309913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:55.311143 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:55.311155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.311878 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.311886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:55.312109 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:55.312847 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:55.312869 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:55.314941 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:55.317774 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:55.317784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:55.317801 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:55.318217 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:55.318564 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:55.318569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:55.318573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:55.318756 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:55.319265 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:55.319274 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:55.319613 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 11:33:55.319862 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 11:33:55.320445 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":319,"threads_started":1}
I20260929 11:33:55.323311 14632 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36939
I20260929 11:33:55.323613 14632 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:55.324329 14632 heartbeater.cc:511] Master 127.6.9.254:36939 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:33:55.326514 14443 ts_manager.cc:194] Registered new tserver with Master: 65292315cdf34a2eaca8e733b1d32de7 (127.6.9.193:44453)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
I20260929 11:33:55.328101 14443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:43965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:33:55.353029 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:55.681130 14638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:55.681445 14638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:55.681571 14638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:55.688854 14638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:55.688982 14638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:55.689129 14638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:55.698776 14638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:55.700057 14638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:55.701349 14638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:55.714438 14643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:55.714421 14644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:55.714474 14646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:55.715291 14638 server_base.cc:1034] running on GCE node
I20260929 11:33:55.715914 14638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:55.716897 14638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:55.718093 14638 hybrid_clock.cc:648] HybridClock initialized: now 1790681635718017 us; error 68 us; skew 500 ppm
I20260929 11:33:55.718417 14638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:55 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:55.721513 14638 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:33:55.723024 14638 webserver.cc:460] Webserver started at http://127.6.9.194:44889/ using document root <none> and password file <none>
I20260929 11:33:55.723783 14638 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:55.723913 14638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:55.724205 14638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:55.727109 14638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "cf51d6486dc2420c8433d5781c809314"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "649bdfec2e642c6c2b71647ca3c60e77"
server_key_iv: "1f95eab4db04c9997626368846110a83"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.727877 14638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "cf51d6486dc2420c8433d5781c809314"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "649bdfec2e642c6c2b71647ca3c60e77"
server_key_iv: "1f95eab4db04c9997626368846110a83"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.733435 14638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:33:55.737187 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.738763 14638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:33:55.738950 14638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "cf51d6486dc2420c8433d5781c809314"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "649bdfec2e642c6c2b71647ca3c60e77"
server_key_iv: "1f95eab4db04c9997626368846110a83"
server_key_version: "encryptionkey@0"
I20260929 11:33:55.739284 14638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:55.755084 14638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:55.759810 14638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:55.760149 14638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:55.761227 14638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:55.762648 14638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:55.762729 14638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.762910 14638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:55.763139 14638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:55.793622 14638 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:36845
I20260929 11:33:55.793684 14765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:36845 every 8 connection(s)
I20260929 11:33:55.794896 14638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:55.796977 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14638
I20260929 11:33:55.797132 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:33:55.797637 6183 external_mini_cluster.cc:1428] Setting key 4eb1f5c6044e0646015b4e5689ec245d
Sep 29 11:33:55 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:55.815407 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:55.797646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59475 (local address 127.6.9.254:36939)
0929 11:33:55.797849 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:55.797855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:55.799173 (+ 1318us) server_negotiation.cc:407] Connection header received
0929 11:33:55.801155 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:55.801160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:55.801251 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:55.801338 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:55.802943 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.803654 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.805319 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.805753 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.805867 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:55.809614 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:55.809619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:55.809622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:55.809662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:55.812137 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:55.813024 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:55.813028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:55.813030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:55.813167 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:55.813614 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:55.813618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:55.813620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:55.813889 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:55.814031 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:55.814770 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 11:33:55.815066 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:33:55.816395 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:55.798207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:59475)
0929 11:33:55.799020 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:55.799069 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:55.800873 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:55.801605 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:55.801614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:55.802145 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:55.802753 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:55.802772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.803815 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.803819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:55.805160 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:55.805174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:55.805983 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:55.805990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:55.806226 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:55.806880 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:55.806904 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:55.809346 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:55.812328 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:55.812339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:55.812356 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:55.812889 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:55.813289 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:55.813294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:55.813296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:55.813498 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:55.814014 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:55.814023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:55.814392 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 11:33:55.814635 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 11:33:55.815218 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20260929 11:33:55.817945 14766 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36939
I20260929 11:33:55.818228 14766 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:33:55.819042 14766 heartbeater.cc:511] Master 127.6.9.254:36939 requested a full tablet report, sending...
I20260929 11:33:55.820596 14443 ts_manager.cc:194] Registered new tserver with Master: cf51d6486dc2420c8433d5781c809314 (127.6.9.194:36845)
I20260929 11:33:55.821502 14443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:59475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:33:55.844470 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:56.163177 14771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:56.163480 14771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:56.163614 14771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:56.170989 14771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:56.171139 14771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:56.171296 14771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:56.181419 14771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:56.182760 14771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:56.184124 14771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:56.197259 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:56.197261 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:56.197294 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:56.198241 14771 server_base.cc:1034] running on GCE node
I20260929 11:33:56.198920 14771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:56.199955 14771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:56.201184 14771 hybrid_clock.cc:648] HybridClock initialized: now 1790681636201131 us; error 46 us; skew 500 ppm
I20260929 11:33:56.201514 14771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:56 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681636, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:56.204648 14771 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:33:56.206213 14771 webserver.cc:460] Webserver started at http://127.6.9.195:40009/ using document root <none> and password file <none>
I20260929 11:33:56.206899 14771 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:56.207017 14771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:56.207338 14771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:33:56.210333 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "e7a00fc90f074448b472219edc712244"
format_stamp: "Formatted at 2026-09-29 11:33:56 on dist-test-slave-c0l9"
server_key: "38382eddebb13e96f4c87223890c2b66"
server_key_iv: "466b57577ad48b8136015d4ac7b086bf"
server_key_version: "encryptionkey@0"
I20260929 11:33:56.211124 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a00fc90f074448b472219edc712244"
format_stamp: "Formatted at 2026-09-29 11:33:56 on dist-test-slave-c0l9"
server_key: "38382eddebb13e96f4c87223890c2b66"
server_key_iv: "466b57577ad48b8136015d4ac7b086bf"
server_key_version: "encryptionkey@0"
I20260929 11:33:56.216588 14771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:33:56.220570 14786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:56.222069 14771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:56.222244 14771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "e7a00fc90f074448b472219edc712244"
format_stamp: "Formatted at 2026-09-29 11:33:56 on dist-test-slave-c0l9"
server_key: "38382eddebb13e96f4c87223890c2b66"
server_key_iv: "466b57577ad48b8136015d4ac7b086bf"
server_key_version: "encryptionkey@0"
I20260929 11:33:56.222450 14771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:56.237404 14771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:56.241947 14771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:56.242234 14771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:56.243362 14771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:56.244899 14771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:56.244977 14771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:56.245100 14771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:56.245157 14771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:56.275800 14771 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:44441
I20260929 11:33:56.275869 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:44441 every 8 connection(s)
I20260929 11:33:56.277145 14771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:56.285111 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14771
I20260929 11:33:56.285257 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:33:56.285681 6183 external_mini_cluster.cc:1428] Setting key 121204f7c19b14bcdee25809a326014c
Sep 29 11:33:56 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681636, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:33:56.297659 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:56.279782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54453 (local address 127.6.9.254:36939)
0929 11:33:56.279994 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:56.279999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:56.281262 (+ 1263us) server_negotiation.cc:407] Connection header received
0929 11:33:56.283059 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:56.283064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:56.283136 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:56.283224 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:56.284754 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.285575 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.287529 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.288066 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.288169 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:56.291888 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:56.291893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:56.291897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:56.291933 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:56.294516 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.295301 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.295306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.295308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.295456 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.295919 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.295924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.295926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.296212 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:56.296332 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:56.297212 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 11:33:56.297402 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:33:56.298611 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:56.280306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:54453)
0929 11:33:56.281094 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:56.281142 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:56.282791 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:56.283438 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:56.283446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:56.283962 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:56.284554 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:56.284572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.285736 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.285742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:56.287361 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:56.287378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.288323 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.288332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:56.288638 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:56.289438 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:56.289464 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:56.291648 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:33:56.294686 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:56.294693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:56.294709 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:56.295162 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:56.295591 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:33:56.295595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:33:56.295598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:33:56.295804 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:33:56.296346 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:33:56.296351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:33:56.296812 (+ 461us) client_negotiation.cc:769] Sending connection context
0929 11:33:56.297081 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 11:33:56.297526 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":338,"threads_started":1}
I20260929 11:33:56.300287 14899 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36939
I20260929 11:33:56.300712 14899 heartbeater.cc:464] Registering TS with master...
I20260929 11:33:56.301640 14899 heartbeater.cc:511] Master 127.6.9.254:36939 requested a full tablet report, sending...
I20260929 11:33:56.303462 14443 ts_manager.cc:194] Registered new tserver with Master: e7a00fc90f074448b472219edc712244 (127.6.9.195:44441)
I20260929 11:33:56.304301 14443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:54453
I20260929 11:33:56.314080 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.9.193@KRBTEST.COM; defaulting to no policy
I20260929 11:33:56.331466 14632 heartbeater.cc:503] Master 127.6.9.254:36939 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
Entry for principal oryx/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab.
WARNING: no policy specified for oryx/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
Entry for principal oryx/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab.
WARNING: no policy specified for oryx/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
Entry for principal oryx/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab.
I20260929 11:33:56.447461 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14504
Sep 29 11:33:56 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:33:56.465092 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:56.448528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.6.9.193:44453)
0929 11:33:56.449175 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:56.449192 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:56.449227 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:33:56.449416 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:56.449424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:56.449662 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:56.449827 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:56.451254 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.452309 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.453689 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.454299 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.454420 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:56.458929 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:56.458947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:56.458965 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:56.459019 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:56.462052 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.462861 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.462869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.462871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.463014 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.463415 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.463419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.463421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.463816 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:56.464114 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:56.464434 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 11:33:56.464739 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":299,"threads_started":1}
I20260929 11:33:56.466316 14627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:56.548620 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14504
I20260929 11:33:56.561872 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14638
Sep 29 11:33:56 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:33:56.577936 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:56.562742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.6.9.194:36845)
0929 11:33:56.563227 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:56.563234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:56.563258 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:33:56.563326 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:56.563332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:56.563533 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:56.563658 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:56.564857 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.565837 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.567170 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.567728 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.567839 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:56.572043 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:56.572059 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:56.572076 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:56.572118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:56.575031 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.575852 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.575861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.575863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.576010 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.576335 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.576337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.576340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.576764 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:56.577002 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:56.577313 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 11:33:56.577573 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
I20260929 11:33:56.579200 14761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:56.661636 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14638
I20260929 11:33:56.674504 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14771
Sep 29 11:33:56 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:33:56.691619 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:56.675637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.6.9.195:44441)
0929 11:33:56.676310 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:56.676317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:56.676346 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:33:56.676407 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:56.676413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:56.676640 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:56.676806 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:56.678096 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.679147 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.680576 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:56.681262 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:56.681365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:56.685891 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:56.685905 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:56.685921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:56.685961 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:56.688762 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.689648 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.689655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.689657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.689807 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:56.690188 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:56.690191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:56.690193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:56.690545 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:56.690789 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:56.691101 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 11:33:56.691308 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":336,"threads_started":1}
I20260929 11:33:56.693145 14894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:56.775681 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14771
I20260929 11:33:56.788638 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14411
I20260929 11:33:56.790002 14473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:33:56.893558 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14411
I20260929 11:33:56.910856 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36939
--webserver_interface=127.6.9.254
--webserver_port=35691
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:36939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:33:57.243462 14916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:57.243813 14916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:57.243963 14916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:57.251647 14916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:33:57.251793 14916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:57.251876 14916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:57.251955 14916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:33:57.252017 14916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:33:57.262302 14916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:36939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=35691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:57.263652 14916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:57.265115 14916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:57.278091 14921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:57.278055 14922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:57.278146 14924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:57.278517 14916 server_base.cc:1034] running on GCE node
I20260929 11:33:57.279719 14916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:57.281315 14916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:57.282560 14916 hybrid_clock.cc:648] HybridClock initialized: now 1790681637282500 us; error 45 us; skew 500 ppm
I20260929 11:33:57.282950 14916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:57.286357 14916 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:33:57.288072 14916 webserver.cc:460] Webserver started at http://127.6.9.254:35691/ using document root <none> and password file <none>
I20260929 11:33:57.288843 14916 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:57.288985 14916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:57.295002 14916 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:33:57.298820 14931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:57.300566 14916 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:33:57.300840 14916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "e0ad4574c5a84da6ab52f47258764b9f"
format_stamp: "Formatted at 2026-09-29 11:33:54 on dist-test-slave-c0l9"
server_key: "c168526690fe9802f25baba96a095db6"
server_key_iv: "b5fc47b6bf50bcd9f257ec5f87b30a56"
server_key_version: "encryptionkey@0"
I20260929 11:33:57.302212 14916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:57.314270 14916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:57.318806 14916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:57.319088 14916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:57.337616 14916 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:36939
I20260929 11:33:57.337656 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:36939 every 8 connection(s)
I20260929 11:33:57.339170 14916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:33:57.342366 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14916
I20260929 11:33:57.343451 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:44453
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:36939
--webserver_port=40315
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:33:57.348160 14983 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:33:57.352591 14983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Bootstrap starting.
I20260929 11:33:57.372136 14983 log.cc:824] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:33:57.381805 14983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:33:57.382587 14983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Bootstrap complete.
I20260929 11:33:57.393344 14983 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:57.394445 14983 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ad4574c5a84da6ab52f47258764b9f, State: Initialized, Role: FOLLOWER
I20260929 11:33:57.395160 14983 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:57.395417 14983 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:33:57.395539 14983 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:33:57.395704 14983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:33:57.400243 14983 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:57.400869 14983 leader_election.cc:302] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ad4574c5a84da6ab52f47258764b9f; no voters:
I20260929 11:33:57.401504 14983 leader_election.cc:288] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:33:57.401660 14987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:33:57.402774 14987 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [term 2 LEADER]: Becoming Leader. State: Replica: e0ad4574c5a84da6ab52f47258764b9f, State: Running, Role: LEADER
I20260929 11:33:57.403419 14987 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } }
I20260929 11:33:57.403765 14983 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:33:57.405925 14989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ad4574c5a84da6ab52f47258764b9f. Latest consensus state: current_term: 2 leader_uuid: "e0ad4574c5a84da6ab52f47258764b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } } }
I20260929 11:33:57.406821 14989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:57.406674 14988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e0ad4574c5a84da6ab52f47258764b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ad4574c5a84da6ab52f47258764b9f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36939 } } }
I20260929 11:33:57.407608 14988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f [sys.catalog]: This master's current role is: LEADER
I20260929 11:33:57.414939 14993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:33:57.421839 14993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:33:57.423539 14993 catalog_manager.cc:1259] Loaded cluster ID: 5c9d09da6b764301a8ffe7d32d1329d1
I20260929 11:33:57.423749 14993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:33:57.428046 14993 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:33:57.429858 14993 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e0ad4574c5a84da6ab52f47258764b9f: Loaded TSK: 0
I20260929 11:33:57.431713 14993 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:33:57.730783 14985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:57.731165 14985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:57.731302 14985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:57.738981 14985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:57.739116 14985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:57.739271 14985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:33:57.750344 14985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.193.keytab
--principal=oryx/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:44453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=40315
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:57.751749 14985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:57.753135 14985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:57.768778 15011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:57.769093 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:57.768748 15010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:57.769984 14985 server_base.cc:1034] running on GCE node
I20260929 11:33:57.770874 14985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:57.772276 14985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:57.773586 14985 hybrid_clock.cc:648] HybridClock initialized: now 1790681637773505 us; error 68 us; skew 500 ppm
I20260929 11:33:57.774012 14985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681637, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:57.778779 14985 init.cc:377] Logged in from keytab as oryx/127.6.9.193@KRBTEST.COM (short username oryx)
I20260929 11:33:57.781008 14985 webserver.cc:460] Webserver started at http://127.6.9.193:40315/ using document root <none> and password file <none>
I20260929 11:33:57.782004 14985 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:57.782195 14985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:57.790076 14985 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:33:57.794435 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:57.796413 14985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:33:57.796657 14985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "65292315cdf34a2eaca8e733b1d32de7"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "8dc22db4f2a4bd4844e4296735c23590"
server_key_iv: "c89192ced9ec5e32debbf1f30334e6f0"
server_key_version: "encryptionkey@0"
I20260929 11:33:57.798144 14985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:57.821712 14985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:57.827764 14985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:57.828192 14985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:57.829543 14985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:57.831310 14985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:57.831429 14985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:57.831542 14985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:57.831610 14985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:57.863003 14985 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:44453
I20260929 11:33:57.863055 15132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:44453 every 8 connection(s)
I20260929 11:33:57.864475 14985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:33:57.869313 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 14985
I20260929 11:33:57.869915 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:36845
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:36939
--webserver_port=44889
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:57.885489 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:57.868246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:51149)
0929 11:33:57.870053 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:57.870107 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:57.872114 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:57.873812 (+ 1698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:57.873824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:57.874476 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:57.875074 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.875095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.876775 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.876780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.878140 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.878155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.879292 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.879299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.879566 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.880256 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:57.880285 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:57.884038 (+ 3753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1482,"thread_start_us":1267,"threads_started":1}
W20260929 11:33:57.885571 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:57.868042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:51149 (local address 127.6.9.254:36939)
0929 11:33:57.869093 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:57.869110 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:57.870579 (+ 1469us) server_negotiation.cc:407] Connection header received
0929 11:33:57.872831 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:57.872846 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:57.873238 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:57.873612 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:57.875307 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.876587 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.878301 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.879026 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.879213 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.884514 (+ 5301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:51149: BlockingRecv error: recv got EOF from 127.6.9.193:51149 (error 108)
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":406,"threads_started":1}
W20260929 11:33:57.886427 15133 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:36939 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.9.254:36939: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:57.896951 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:57.887089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:38471)
0929 11:33:57.887288 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:57.887310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:57.887445 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:57.887764 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:57.887769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:57.888214 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:57.888433 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.888448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.889566 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.889572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.891116 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.891137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.891893 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.891900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.892086 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.892730 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:57.892754 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:57.896499 (+ 3745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 11:33:57.896975 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:57.887316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:38471 (local address 127.6.9.254:36939)
0929 11:33:57.887483 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:57.887488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:57.887505 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:33:57.887558 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:57.887562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:57.887622 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:57.887699 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:57.888588 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.889390 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.891312 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.891737 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.891840 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.896807 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:38471: BlockingRecv error: recv got EOF from 127.6.9.193:38471 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:57 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:57.907023 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:57.897806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:35359)
0929 11:33:57.897991 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:57.898010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:57.898115 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:57.898492 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:57.898496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:57.898814 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:57.898993 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.899003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.900481 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.900486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.901569 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:57.901585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.902510 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.902515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:57.902648 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.903267 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:57.903318 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:57.906774 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 11:33:57.907305 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:57.897933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:35359 (local address 127.6.9.254:36939)
0929 11:33:57.898115 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:57.898122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:57.898142 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:33:57.898223 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:57.898228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:57.898321 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:57.898416 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:57.899159 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.900308 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.901727 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:57.902346 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:57.902469 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:57.907034 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:35359: BlockingRecv error: recv got EOF from 127.6.9.193:35359 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 11:33:57.907542 15133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:33:58.211032 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:58.211344 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:58.211472 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:58.219062 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:58.219199 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:58.219347 15137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:33:58.229899 15137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.194.keytab
--principal=oryx/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:36845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=44889
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:58.231240 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:58.232494 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:58.245289 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:58.245324 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:58.245258 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:58.246397 15137 server_base.cc:1034] running on GCE node
I20260929 11:33:58.247155 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:58.248196 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:58.249414 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790681638249342 us; error 60 us; skew 500 ppm
I20260929 11:33:58.249742 15137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681638, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:58.256487 15137 init.cc:377] Logged in from keytab as oryx/127.6.9.194@KRBTEST.COM (short username oryx)
I20260929 11:33:58.258086 15137 webserver.cc:460] Webserver started at http://127.6.9.194:44889/ using document root <none> and password file <none>
I20260929 11:33:58.258839 15137 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:58.258981 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:58.265002 15137 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:33:58.268112 15152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.269667 15137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:33:58.269872 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "cf51d6486dc2420c8433d5781c809314"
format_stamp: "Formatted at 2026-09-29 11:33:55 on dist-test-slave-c0l9"
server_key: "649bdfec2e642c6c2b71647ca3c60e77"
server_key_iv: "1f95eab4db04c9997626368846110a83"
server_key_version: "encryptionkey@0"
I20260929 11:33:58.270879 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:58.296109 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:58.300458 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:58.300746 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:58.301751 15137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:58.303136 15137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:58.303262 15137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.303366 15137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:58.303431 15137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.334618 15137 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:36845
I20260929 11:33:58.334689 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:36845 every 8 connection(s)
I20260929 11:33:58.336035 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:33:58.342327 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15137
I20260929 11:33:58.343039 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:44441
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:36939
--webserver_port=40009
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:33:58.352533 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.338806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:37233 (local address 127.6.9.254:36939)
0929 11:33:58.338972 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.338976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.340474 (+ 1498us) server_negotiation.cc:407] Connection header received
0929 11:33:58.342410 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.342416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.342507 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.342597 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.344385 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.345155 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.346745 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.347223 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.347336 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.352294 (+ 4958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:37233: BlockingRecv error: recv got EOF from 127.6.9.194:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:33:58.353130 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.339407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:37233)
0929 11:33:58.340281 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.340322 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.342111 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.342947 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.342957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.343593 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.344191 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.344211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.345328 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.345332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.346589 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.346603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.347453 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.347461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.347684 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.348333 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.348357 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.352034 (+ 3677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":369,"threads_started":1}
W20260929 11:33:58.353924 15265 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:36939 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.9.254:36939: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:58.363435 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.354549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:52913)
0929 11:33:58.354714 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.354733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.354845 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.355219 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.355223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.355606 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.355780 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.355791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.356811 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.356816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.358078 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.358094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.358887 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.358893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.359073 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.359700 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.359724 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.363138 (+ 3414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:33:58.363642 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.354686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:52913 (local address 127.6.9.254:36939)
0929 11:33:58.354828 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.354832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.354850 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:33:58.354955 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.354959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.355019 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.355209 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.355925 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.356654 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.358242 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.358723 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.358837 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.363418 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:52913: BlockingRecv error: recv got EOF from 127.6.9.194:52913 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:58.373055 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.364230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:55767)
0929 11:33:58.364504 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.364522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.364609 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.365004 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.365008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.365305 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.365462 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.365470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.366884 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.366889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.367959 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.367973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.369003 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.369008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.369141 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.369759 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.369777 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.372742 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 11:33:58.373106 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.364436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:55767 (local address 127.6.9.254:36939)
0929 11:33:58.364623 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.364629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.364651 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:33:58.364768 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.364774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.364853 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.364956 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.365603 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.366729 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.368125 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.368836 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.368982 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.372942 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:55767: BlockingRecv error: recv got EOF from 127.6.9.194:55767 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 11:33:58.373395 15265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:33:58.680244 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:33:58.680562 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:33:58.680688 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:33:58.688205 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:33:58.688359 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:33:58.688478 15268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:33:58.698844 15268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/krb5kdc/oryx_127.6.9.195.keytab
--principal=oryx/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:44441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=40009
--tserver_master_addrs=127.6.9.254:36939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:33:58.700160 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:33:58.701491 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:33:58.714720 15274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:58.714772 15273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:33:58.714749 15276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:33:58.716120 15268 server_base.cc:1034] running on GCE node
I20260929 11:33:58.716904 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:33:58.717999 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:33:58.719261 15268 hybrid_clock.cc:648] HybridClock initialized: now 1790681638719184 us; error 64 us; skew 500 ppm
I20260929 11:33:58.719657 15268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681638, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:33:58.723119 15268 init.cc:377] Logged in from keytab as oryx/127.6.9.195@KRBTEST.COM (short username oryx)
I20260929 11:33:58.724843 15268 webserver.cc:460] Webserver started at http://127.6.9.195:40009/ using document root <none> and password file <none>
I20260929 11:33:58.725570 15268 fs_manager.cc:359] Metadata directory not provided
I20260929 11:33:58.725703 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:33:58.731673 15268 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 11:33:58.734579 15283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.736006 15268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:33:58.736160 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "e7a00fc90f074448b472219edc712244"
format_stamp: "Formatted at 2026-09-29 11:33:56 on dist-test-slave-c0l9"
server_key: "38382eddebb13e96f4c87223890c2b66"
server_key_iv: "466b57577ad48b8136015d4ac7b086bf"
server_key_version: "encryptionkey@0"
I20260929 11:33:58.737177 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:33:58.754817 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:33:58.759191 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:33:58.759447 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:33:58.760506 15268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:33:58.761981 15268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:33:58.762072 15268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.762156 15268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:33:58.762212 15268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:33:58.792654 15268 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:44441
I20260929 11:33:58.792750 15395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:44441 every 8 connection(s)
I20260929 11:33:58.794090 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:33:58.799201 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15268
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:33:58.812326 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.796616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:38159 (local address 127.6.9.254:36939)
0929 11:33:58.796827 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.796832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.798055 (+ 1223us) server_negotiation.cc:407] Connection header received
0929 11:33:58.799974 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.799981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.800067 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.800160 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.802193 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.803262 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.805439 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.806119 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.806245 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.812091 (+ 5846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:38159: BlockingRecv error: recv got EOF from 127.6.9.195:38159 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:33:58.813270 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.797155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:38159)
0929 11:33:58.797886 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.797929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.799668 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.800630 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.800645 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.801367 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.801998 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.802020 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.803445 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.803451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.805271 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.805286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.806352 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.806358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.806685 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.807639 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.807673 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.811849 (+ 4176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
W20260929 11:33:58.814105 15396 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:36939 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.9.254:36939: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
I20260929 11:33:58.817767 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.800235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.6.9.254:36939)
0929 11:33:58.801114 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.801120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.801136 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:33:58.801184 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.801187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.801243 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.801333 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.803164 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.804344 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.806119 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.806742 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.806868 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.809907 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:33:58.809918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:33:58.809938 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:33:58.809980 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:33:58.813832 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:58.814763 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:58.814773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:58.814782 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:58.815020 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:33:58.815483 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:33:58.815487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:33:58.815489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:33:58.816099 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:33:58.816436 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 11:33:58.816969 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 11:33:58.817370 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":753,"threads_started":1}
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:58.823192 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.814731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:57891)
0929 11:33:58.814894 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.814913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.815006 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.815344 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.815349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.815696 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.815859 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.815868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.817277 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.817283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.818441 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.818456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.819189 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.819195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.819364 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.819919 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.819937 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.822984 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:33:58.823408 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.814798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:57891 (local address 127.6.9.254:36939)
0929 11:33:58.814966 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.814972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.814988 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:33:58.815103 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.815109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.815191 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.815262 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.815998 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.817109 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.818585 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.819042 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.819152 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.823190 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:57891: BlockingRecv error: recv got EOF from 127.6.9.195:57891 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:58.831568 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.823862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:40763)
0929 11:33:58.824068 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.824087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.824199 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.824590 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.824593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.824923 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.825083 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.825092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.826089 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.826093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.827090 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.827103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.827804 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.827809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.827945 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.828498 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.828516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.831332 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:33:58.831866 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.823945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:40763 (local address 127.6.9.254:36939)
0929 11:33:58.824285 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.824291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.824310 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:33:58.824363 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.824366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.824436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.824513 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.825213 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.825937 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.827233 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.827656 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.827748 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.831579 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:40763: BlockingRecv error: recv got EOF from 127.6.9.195:40763 (error 108)
Metrics: {"server-negotiator.queue_time_us":233}
W20260929 11:33:58.831882 15396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:58 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:58.917035 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:58.908239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:52569)
0929 11:33:58.908806 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:58.908828 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:58.908980 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:58.909458 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:58.909463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:58.909934 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:58.910137 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.910147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.911206 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.911211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.912189 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:58.912202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.913040 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.913047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:58.913208 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.913795 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:58.913814 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:58.916794 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
W20260929 11:33:58.917272 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:58.908285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:52569 (local address 127.6.9.254:36939)
0929 11:33:58.908510 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:58.908518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:58.908918 (+ 400us) server_negotiation.cc:407] Connection header received
0929 11:33:58.909161 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:58.909166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:58.909248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:58.909334 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:58.910285 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.911023 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.912344 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:58.912852 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:58.912969 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:58.917039 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:52569: BlockingRecv error: recv got EOF from 127.6.9.193:52569 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:59.383329 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:59.374011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:59509)
0929 11:33:59.374522 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:59.374537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:59.374661 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:59.375000 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:59.375004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:59.375542 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:59.375747 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.375766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.377006 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.377014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.378273 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.378288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.379083 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.379089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.379250 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.379916 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:59.379937 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:59.383004 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":261,"threads_started":1}
W20260929 11:33:59.383486 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:59.374148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59509 (local address 127.6.9.254:36939)
0929 11:33:59.374402 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:59.374408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:59.374617 (+ 209us) server_negotiation.cc:407] Connection header received
0929 11:33:59.374764 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:59.374768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:59.374840 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:59.374921 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:59.375941 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.376790 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.378440 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.378887 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.378992 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.383255 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:59509: BlockingRecv error: recv got EOF from 127.6.9.194:59509 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:59.841099 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:59.832476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:56057)
0929 11:33:59.833004 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:59.833021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:59.833142 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:59.833434 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:59.833437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:59.833882 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:59.834084 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.834093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.835126 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.835131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.836092 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.836105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.836813 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.836817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.836957 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.837684 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:59.837704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:59.840771 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
W20260929 11:33:59.841256 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:59.832527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:56057 (local address 127.6.9.254:36939)
0929 11:33:59.832771 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:59.832776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:59.833090 (+ 314us) server_negotiation.cc:407] Connection header received
0929 11:33:59.833233 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:59.833237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:59.833317 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:59.833399 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:59.834228 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.834957 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.836229 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.836639 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.836767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.840986 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:56057: BlockingRecv error: recv got EOF from 127.6.9.195:56057 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:33:59 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:33:59.926230 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:33:59.917979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:46551)
0929 11:33:59.918505 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 11:33:59.918520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:33:59.918631 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:33:59.918978 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:33:59.918982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:33:59.919369 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:33:59.919525 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.919534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.920564 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.920569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.921540 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:33:59.921555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.922291 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.922296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:33:59.922430 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.923009 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:33:59.923028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:33:59.925947 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
W20260929 11:33:59.926513 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:33:59.918071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:46551 (local address 127.6.9.254:36939)
0929 11:33:59.918272 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:33:59.918276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:33:59.918583 (+ 307us) server_negotiation.cc:407] Connection header received
0929 11:33:59.918765 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:33:59.918769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:33:59.918850 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:33:59.918944 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:33:59.919660 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.920419 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.921687 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:33:59.922124 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:33:59.922231 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:33:59.926238 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:46551: BlockingRecv error: recv got EOF from 127.6.9.193:46551 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:00.392812 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:00.384383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:56359)
0929 11:34:00.384948 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:00.384966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:00.385107 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:00.385516 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:00.385521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:00.386065 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:00.386241 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.386252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.387233 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.387237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.388138 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.388150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.388917 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.388922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.389036 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.389620 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:00.389639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:00.392527 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":300,"threads_started":1}
W20260929 11:34:00.393009 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:00.384504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56359 (local address 127.6.9.254:36939)
0929 11:34:00.384768 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:00.384773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:00.385034 (+ 261us) server_negotiation.cc:407] Connection header received
0929 11:34:00.385239 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:00.385243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:00.385336 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:00.385433 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:00.386391 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.387103 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.388287 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.388767 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.388867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.392794 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:56359: BlockingRecv error: recv got EOF from 127.6.9.194:56359 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:00.850572 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:00.842058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:52373)
0929 11:34:00.842563 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:00.842583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:00.842733 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:00.843152 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:00.843157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:00.843646 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:00.843827 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.843841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.845073 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.845079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.845992 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.846005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.846695 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.846700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.846821 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.847427 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:00.847462 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:00.850231 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
W20260929 11:34:00.850739 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:00.842335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:52373 (local address 127.6.9.254:36939)
0929 11:34:00.842542 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:00.842547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:00.842662 (+ 115us) server_negotiation.cc:407] Connection header received
0929 11:34:00.842891 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:00.842895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:00.842998 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:00.843103 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:00.844005 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.844921 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.846125 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.846544 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.846638 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.850478 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:52373: BlockingRecv error: recv got EOF from 127.6.9.195:52373 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:00 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:00.935554 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:00.927321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:49961)
0929 11:34:00.927819 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:00.927839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:00.927965 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:00.928287 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:00.928291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:00.928627 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:00.928859 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.928871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.930087 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.930092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.931264 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:00.931277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.931962 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.931966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:00.932084 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.932615 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:00.932631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:00.935306 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
W20260929 11:34:00.935844 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:00.927473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:49961 (local address 127.6.9.254:36939)
0929 11:34:00.927631 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:00.927636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:00.927926 (+ 290us) server_negotiation.cc:407] Connection header received
0929 11:34:00.928063 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:00.928067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:00.928133 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:00.928244 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:00.929058 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.929934 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.931425 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:00.931849 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:00.931962 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:00.935550 (+ 3588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:49961: BlockingRecv error: recv got EOF from 127.6.9.193:49961 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:01.401934 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:01.393835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:56405)
0929 11:34:01.394295 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:01.394312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:01.394435 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:01.394778 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:01.394781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:01.395132 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:01.395303 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.395312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.396343 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.396347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.397251 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.397262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.397935 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.397939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.398055 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.398656 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:01.398674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:01.401675 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
W20260929 11:34:01.402077 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:01.393962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56405 (local address 127.6.9.254:36939)
0929 11:34:01.394192 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:01.394199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:01.394392 (+ 193us) server_negotiation.cc:407] Connection header received
0929 11:34:01.394540 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:01.394544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:01.394612 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:01.394702 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:01.395451 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.396226 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.397408 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.397821 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.397923 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.401878 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:56405: BlockingRecv error: recv got EOF from 127.6.9.194:56405 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:01.859750 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:01.851646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:50931)
0929 11:34:01.852115 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:01.852131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:01.852253 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:01.852577 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:01.852581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:01.852928 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:01.853094 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.853102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.854127 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.854131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.855051 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.855063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.855785 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.855790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.855924 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.856590 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:01.856607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:01.859501 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":247,"threads_started":1}
W20260929 11:34:01.859894 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:01.851775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:50931 (local address 127.6.9.254:36939)
0929 11:34:01.851997 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:01.852004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:01.852206 (+ 202us) server_negotiation.cc:407] Connection header received
0929 11:34:01.852382 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:01.852386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:01.852465 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:01.852545 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:01.853256 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.853996 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.855205 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.855641 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.855769 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.859696 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:50931: BlockingRecv error: recv got EOF from 127.6.9.195:50931 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:01 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:01.944871 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:01.936471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:51243)
0929 11:34:01.936960 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:01.936979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:01.937109 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:01.937390 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:01.937393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:01.937760 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:01.937935 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.937945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.938966 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.938971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.939945 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:01.939959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.940788 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.940794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:01.940944 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.941563 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:01.941582 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:01.944606 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
W20260929 11:34:01.945097 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:01.936572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:51243 (local address 127.6.9.254:36939)
0929 11:34:01.936831 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:01.936837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:01.937057 (+ 220us) server_negotiation.cc:407] Connection header received
0929 11:34:01.937206 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:01.937210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:01.937273 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:01.937352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:01.938058 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.938786 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.940113 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:01.940607 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:01.940741 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:01.944855 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:51243: BlockingRecv error: recv got EOF from 127.6.9.193:51243 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:02.411813 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:02.402776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:59679)
0929 11:34:02.403206 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:02.403222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:02.403360 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:02.403694 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:02.403697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:02.404111 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:02.404274 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.404284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.405507 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.405511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.406569 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.406583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.407475 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.407481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.407660 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.408276 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:02.408301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:02.411496 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":233,"threads_started":1}
W20260929 11:34:02.411935 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:02.402918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59679 (local address 127.6.9.254:36939)
0929 11:34:02.403147 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:02.403154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:02.403299 (+ 145us) server_negotiation.cc:407] Connection header received
0929 11:34:02.403507 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:02.403511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:02.403578 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:02.403664 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:02.404499 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.405333 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.406713 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.407303 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.407454 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.411722 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:59679: BlockingRecv error: recv got EOF from 127.6.9.194:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:02.869287 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:02.860586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:60233)
0929 11:34:02.861055 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:02.861072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:02.861192 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:02.861460 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:02.861463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:02.861813 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:02.861971 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.861979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.862969 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.862975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.864477 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.864494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.865320 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.865326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.865539 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.866133 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:02.866155 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:02.868973 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":267,"threads_started":1}
W20260929 11:34:02.869509 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:02.860742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:60233 (local address 127.6.9.254:36939)
0929 11:34:02.860940 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:02.860946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:02.861141 (+ 195us) server_negotiation.cc:407] Connection header received
0929 11:34:02.861275 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:02.861278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:02.861341 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:02.861425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:02.862078 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.862824 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.864644 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.865162 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.865261 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.869219 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:60233: BlockingRecv error: recv got EOF from 127.6.9.195:60233 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:02 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:02.953948 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:02.945788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:57253)
0929 11:34:02.946432 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:02.946449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:02.946560 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:02.946825 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:02.946828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:02.947211 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:02.947377 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.947405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.948452 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.948456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.949404 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:02.949417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.950167 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.950174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:02.950291 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.950863 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:02.950880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:02.953699 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":237,"threads_started":1}
W20260929 11:34:02.954151 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:02.945848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:57253 (local address 127.6.9.254:36939)
0929 11:34:02.946083 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:02.946089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:02.946521 (+ 432us) server_negotiation.cc:407] Connection header received
0929 11:34:02.946647 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:02.946650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:02.946716 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:02.946796 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:02.947567 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.948306 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.949572 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:02.950039 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:02.950149 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:02.953957 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:57253: BlockingRecv error: recv got EOF from 127.6.9.193:57253 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:03.426429 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:03.412991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:36859)
0929 11:34:03.413508 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:03.413527 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:03.413653 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:03.414019 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:03.414024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:03.414501 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:03.414709 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.414723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.415714 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.415720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.416629 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.416642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.417416 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.417420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.417542 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.423201 (+ 5659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:03.423224 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:03.426106 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":344,"threads_started":1}
W20260929 11:34:03.426671 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:03.413440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:36859 (local address 127.6.9.254:36939)
0929 11:34:03.413606 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:03.413612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:03.413630 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:34:03.413794 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:03.413800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:03.413877 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:03.413965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:03.414825 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.415576 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.416781 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.417290 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.417413 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.426416 (+ 9003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:36859: BlockingRecv error: recv got EOF from 127.6.9.194:36859 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:03.878736 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:03.870395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:52505)
0929 11:34:03.870888 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:03.870906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:03.871025 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:03.871377 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:03.871380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:03.871804 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:03.871969 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.871978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.873025 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.873028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.874067 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.874080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.874871 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.874877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.875075 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.875701 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:03.875720 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:03.878407 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
W20260929 11:34:03.878887 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:03.870551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:52505 (local address 127.6.9.254:36939)
0929 11:34:03.870707 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:03.870712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:03.871013 (+ 301us) server_negotiation.cc:407] Connection header received
0929 11:34:03.871157 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:03.871161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:03.871238 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:03.871319 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:03.872126 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.872910 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.874192 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.874704 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.874821 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.878649 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:52505: BlockingRecv error: recv got EOF from 127.6.9.195:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:03 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:03.963820 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:03.954963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:54217)
0929 11:34:03.955491 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:03.955508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:03.955634 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:03.955989 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:03.955993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:03.956414 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:03.956569 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.956579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.957708 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.957713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.958656 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:03.958670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.959447 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.959452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:03.959616 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.960406 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:03.960431 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:03.963595 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
W20260929 11:34:03.964108 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:03.955103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:54217 (local address 127.6.9.254:36939)
0929 11:34:03.955273 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:03.955277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:03.955620 (+ 343us) server_negotiation.cc:407] Connection header received
0929 11:34:03.955733 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:03.955737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:03.955805 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:03.955904 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:03.956748 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.957511 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.958824 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:03.959278 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:03.959399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:03.963862 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:54217: BlockingRecv error: recv got EOF from 127.6.9.193:54217 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:04.435575 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:04.427443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:58795)
0929 11:34:04.427923 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:04.427939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:04.428070 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:04.428332 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:04.428335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:04.428766 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:04.428926 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.428936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.429916 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.429920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.430840 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.430852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.431541 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.431545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.431671 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.432245 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:04.432263 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:04.435325 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
W20260929 11:34:04.435818 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:04.427592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:58795 (local address 127.6.9.254:36939)
0929 11:34:04.427819 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:04.427825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:04.428011 (+ 186us) server_negotiation.cc:407] Connection header received
0929 11:34:04.428168 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:04.428172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:04.428236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:04.428313 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:04.429059 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.429778 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.430983 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.431413 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.431518 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.435576 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:58795: BlockingRecv error: recv got EOF from 127.6.9.194:58795 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:04.888139 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:04.879807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:58405)
0929 11:34:04.880383 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:04.880399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:04.880541 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:04.880895 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:04.880900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:04.881306 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:04.881452 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.881461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.882468 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.882473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.883395 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.883407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.884139 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.884145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.884308 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.884874 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:04.884891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:04.887916 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":329,"threads_started":1}
W20260929 11:34:04.888414 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:04.879893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:58405 (local address 127.6.9.254:36939)
0929 11:34:04.880132 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:04.880143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:04.880467 (+ 324us) server_negotiation.cc:407] Connection header received
0929 11:34:04.880653 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:04.880658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:04.880761 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:04.880840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:04.881599 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.882330 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.883546 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.883987 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.884098 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.888169 (+ 4071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:58405: BlockingRecv error: recv got EOF from 127.6.9.195:58405 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:04 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:04.973474 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:04.964961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:51385)
0929 11:34:04.965552 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:04.965569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:04.965693 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:04.966069 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:04.966073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:04.966500 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:04.966682 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.966693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.967781 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.967785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.968754 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:04.968770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.969469 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.969475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:04.969635 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.970205 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:04.970220 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:04.973074 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":241,"threads_started":1}
W20260929 11:34:04.973595 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:04.965062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:51385 (local address 127.6.9.254:36939)
0929 11:34:04.965252 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:04.965256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:04.965674 (+ 418us) server_negotiation.cc:407] Connection header received
0929 11:34:04.965806 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:04.965811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:04.965896 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:04.966002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:04.966838 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.967631 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.968899 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:04.969310 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:04.969434 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:04.973350 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:51385: BlockingRecv error: recv got EOF from 127.6.9.193:51385 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:05.446812 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:05.436600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:58401)
0929 11:34:05.437255 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:05.437275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:05.437422 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:05.437958 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:05.437963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:05.438496 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:05.438719 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.438731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.440588 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.440594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.442192 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.442209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.443194 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.443200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.443335 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.443877 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:05.443895 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:05.446562 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":249,"threads_started":1}
W20260929 11:34:05.447006 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:05.436752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:58401 (local address 127.6.9.254:36939)
0929 11:34:05.437059 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:05.437065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:05.437640 (+ 575us) server_negotiation.cc:407] Connection header received
0929 11:34:05.437722 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:05.437728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:05.437825 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:05.437914 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:05.438856 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.439925 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.442357 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.443005 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.443138 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.446763 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:58401: BlockingRecv error: recv got EOF from 127.6.9.194:58401 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:05.897487 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:05.889199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:33943)
0929 11:34:05.889795 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:05.889810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:05.889929 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:05.890216 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:05.890219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:05.890598 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:05.890774 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.890783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.891788 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.891791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.892762 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.892773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.893480 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.893484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.893588 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.894112 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:05.894128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:05.897189 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
W20260929 11:34:05.897642 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:05.889263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:33943 (local address 127.6.9.254:36939)
0929 11:34:05.889475 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:05.889484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:05.889892 (+ 408us) server_negotiation.cc:407] Connection header received
0929 11:34:05.890027 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:05.890031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:05.890099 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:05.890188 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:05.890931 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.891671 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.892894 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.893363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.893468 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.897423 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:33943: BlockingRecv error: recv got EOF from 127.6.9.195:33943 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:05 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:05.982561 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:05.974532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:48681)
0929 11:34:05.975037 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:05.975054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:05.975172 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:05.975539 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:05.975543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:05.975928 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:05.976104 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.976114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.977118 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.977122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.978044 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:05.978056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.978778 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.978783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:05.978899 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.979480 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:05.979499 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:05.982288 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":406,"threads_started":1}
W20260929 11:34:05.982775 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:05.974729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:48681 (local address 127.6.9.254:36939)
0929 11:34:05.974938 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:05.974942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:05.975137 (+ 195us) server_negotiation.cc:407] Connection header received
0929 11:34:05.975349 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:05.975353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:05.975418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:05.975499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:05.976223 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.976985 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.978173 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:05.978631 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:05.978722 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:05.982528 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:48681: BlockingRecv error: recv got EOF from 127.6.9.193:48681 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:06.456627 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:06.447792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:43193)
0929 11:34:06.448287 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:06.448302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:06.448437 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:06.448770 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:06.448774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:06.449197 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:06.449343 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.449352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.450595 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.450599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.451732 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.451747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.452644 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.452650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.452816 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.453425 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:06.453443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:06.456320 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":260,"threads_started":1}
W20260929 11:34:06.456805 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:06.447945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:43193 (local address 127.6.9.254:36939)
0929 11:34:06.448160 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:06.448166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:06.448380 (+ 214us) server_negotiation.cc:407] Connection header received
0929 11:34:06.448565 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:06.448569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:06.448629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:06.448743 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:06.449513 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.450408 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.451874 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.452407 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.452541 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.456556 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:43193: BlockingRecv error: recv got EOF from 127.6.9.194:43193 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:06.907044 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:06.898677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:47127)
0929 11:34:06.899158 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:06.899179 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:06.899325 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:06.899641 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:06.899644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:06.900154 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:06.900381 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.900396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.901524 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.901528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.902444 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.902458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.903153 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.903157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.903303 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.903975 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:06.903994 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:06.906785 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
W20260929 11:34:06.907343 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:06.898755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:47127 (local address 127.6.9.254:36939)
0929 11:34:06.898996 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:06.899002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:06.899271 (+ 269us) server_negotiation.cc:407] Connection header received
0929 11:34:06.899431 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:06.899435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:06.899499 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:06.899588 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:06.900628 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.901372 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.902576 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.903014 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.903107 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.907046 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:47127: BlockingRecv error: recv got EOF from 127.6.9.195:47127 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:06 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:06.992208 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:06.983729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:44693)
0929 11:34:06.984276 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:06.984293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:06.984439 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:06.984841 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:06.984846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:06.985248 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:06.985420 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.985430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.986707 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.986712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.987655 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:06.987667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.988379 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.988384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:06.988554 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.989122 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:06.989139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:06.991952 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
W20260929 11:34:06.992401 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:06.983897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44693 (local address 127.6.9.254:36939)
0929 11:34:06.984159 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:06.984165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:06.984386 (+ 221us) server_negotiation.cc:407] Connection header received
0929 11:34:06.984599 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:06.984606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:06.984706 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:06.984790 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:06.985580 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.986563 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.987795 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:06.988232 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:06.988340 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:06.992180 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:44693: BlockingRecv error: recv got EOF from 127.6.9.193:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:07.466239 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:07.457588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:53489)
0929 11:34:07.458159 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:07.458175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:07.458291 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:07.458661 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:07.458664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:07.459027 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:07.459200 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.459210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.460286 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.460290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.461265 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.461279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.462033 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.462037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.462175 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:07.462777 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:07.462797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:07.465965 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
W20260929 11:34:07.466468 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:07.457721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:53489 (local address 127.6.9.254:36939)
0929 11:34:07.457901 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:07.457909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:07.458257 (+ 348us) server_negotiation.cc:407] Connection header received
0929 11:34:07.458439 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:07.458444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:07.458533 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:07.458620 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:07.459373 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.460155 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.461422 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.461880 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.461993 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:07.466220 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:53489: BlockingRecv error: recv got EOF from 127.6.9.194:53489 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:07.918061 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:07.908101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:53815)
0929 11:34:07.909111 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:07.909130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:07.909297 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:07.909676 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:07.909680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:07.910135 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:07.910331 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.910343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.911522 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.911529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.912542 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.912558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.913399 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.913406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.913600 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:07.914230 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:07.914252 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:07.917708 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":743,"threads_started":1}
W20260929 11:34:07.918136 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:07.908190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:53815 (local address 127.6.9.254:36939)
0929 11:34:07.908459 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:07.908467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:07.909242 (+ 775us) server_negotiation.cc:407] Connection header received
0929 11:34:07.909454 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:07.909458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:07.909528 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:07.909620 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:07.910531 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.911352 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.912742 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.913217 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.913350 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:07.917949 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:53815: BlockingRecv error: recv got EOF from 127.6.9.195:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:07 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:08.002641 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:07.993287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:58337)
0929 11:34:07.993792 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:07.993815 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:07.993960 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:07.994368 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:07.994373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:07.994814 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:07.995024 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.995038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.996543 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.996548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.997549 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:07.997562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.998579 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.998584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:07.998729 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:07.999325 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:07.999343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:08.002268 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
W20260929 11:34:08.002650 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:07.993405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58337 (local address 127.6.9.254:36939)
0929 11:34:07.993599 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:07.993603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:07.993908 (+ 305us) server_negotiation.cc:407] Connection header received
0929 11:34:07.994088 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:07.994095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:07.994185 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:07.994303 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:07.995192 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.996371 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.997697 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:07.998399 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:07.998542 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:08.002441 (+ 3899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:58337: BlockingRecv error: recv got EOF from 127.6.9.193:58337 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 11:34:08 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:08 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:08.479805 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:08.467258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:56287)
0929 11:34:08.467778 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:08.467793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:08.467921 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:08.468225 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:08.468230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:08.468636 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:08.468823 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:08.468834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.469801 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.469806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:08.470726 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:08.470738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.471428 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.471443 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:08.471603 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:08.476464 (+ 4861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:08.476490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:08.479576 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":306,"threads_started":1}
W20260929 11:34:08.480028 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:08.467354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56287 (local address 127.6.9.254:36939)
0929 11:34:08.467569 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:08.467576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:08.467874 (+ 298us) server_negotiation.cc:407] Connection header received
0929 11:34:08.468040 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:08.468045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:08.468104 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:08.468182 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:08.468972 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.469671 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.470860 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.471286 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.471392 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:08.479794 (+ 8402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:56287: BlockingRecv error: recv got EOF from 127.6.9.194:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 11:34:08 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:08 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:08.927855 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:08.919101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:57403)
0929 11:34:08.919623 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:08.919642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:08.919782 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:08.920128 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:08.920132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:08.920524 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:08.920706 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:08.920717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.921971 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.921977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:08.923116 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:08.923132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.923848 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.923852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:08.923985 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:08.924595 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:08.924615 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:08.927535 (+ 2920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260929 11:34:08.928112 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:08.919324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:57403 (local address 127.6.9.254:36939)
0929 11:34:08.919533 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:08.919538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:08.919712 (+ 174us) server_negotiation.cc:407] Connection header received
0929 11:34:08.919871 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:08.919875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:08.919938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:08.920042 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:08.920914 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.921815 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.923272 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:08.923702 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:08.923802 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:08.927818 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:57403: BlockingRecv error: recv got EOF from 127.6.9.195:57403 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:09.012085 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:09.003649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:46653)
0929 11:34:09.004174 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:09.004190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:09.004331 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:09.004711 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:09.004716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:09.005148 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:09.005316 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.005324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.006348 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.006354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.007268 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.007281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.008079 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.008089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.008265 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.008967 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:09.009009 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:09.011832 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
W20260929 11:34:09.012296 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:09.003773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:46653 (local address 127.6.9.254:36939)
0929 11:34:09.004004 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:09.004010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:09.004276 (+ 266us) server_negotiation.cc:407] Connection header received
0929 11:34:09.004467 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:09.004471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:09.004535 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:09.004655 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:09.005464 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.006214 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.007408 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.007908 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.008037 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.012064 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:46653: BlockingRecv error: recv got EOF from 127.6.9.193:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:09.490042 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:09.481067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:44313)
0929 11:34:09.481558 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:09.481574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:09.481705 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:09.482043 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:09.482047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:09.482452 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:09.482658 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.482668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.484101 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.484108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.485203 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.485219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.485994 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.485999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.486132 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.486769 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:09.486789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:09.489781 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":246,"threads_started":1}
W20260929 11:34:09.490249 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:09.481219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:44313 (local address 127.6.9.254:36939)
0929 11:34:09.481448 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:09.481454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:09.481654 (+ 200us) server_negotiation.cc:407] Connection header received
0929 11:34:09.481810 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:09.481814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:09.481880 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:09.481966 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:09.482831 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.483906 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.485353 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.485836 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.485957 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.490032 (+ 4075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:44313: BlockingRecv error: recv got EOF from 127.6.9.194:44313 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:09 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:09.937191 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:09.928855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:49123)
0929 11:34:09.929334 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:09.929350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:09.929477 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:09.929755 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:09.929758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:09.930164 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:09.930345 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.930353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.931339 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.931343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.932286 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:09.932299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.933108 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.933113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:09.933241 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.933934 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:09.933950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:09.936875 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
W20260929 11:34:09.937350 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:09.928938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:49123 (local address 127.6.9.254:36939)
0929 11:34:09.929118 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:09.929123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:09.929457 (+ 334us) server_negotiation.cc:407] Connection header received
0929 11:34:09.929573 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:09.929577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:09.929643 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:09.929743 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:09.930458 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.931196 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.932442 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:09.932917 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:09.933023 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:09.937150 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:49123: BlockingRecv error: recv got EOF from 127.6.9.195:49123 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:10.026723 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:10.013162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:60075)
0929 11:34:10.013690 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:10.013707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:10.013821 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:10.014158 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:10.014162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:10.014565 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:10.014732 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.014742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.015826 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.015831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.016886 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.016902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.017760 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.017765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.017913 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.022655 (+ 4742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:10.022691 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:10.026476 (+ 3785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
W20260929 11:34:10.026942 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:10.013173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:60075 (local address 127.6.9.254:36939)
0929 11:34:10.013362 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:10.013368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:10.013802 (+ 434us) server_negotiation.cc:407] Connection header received
0929 11:34:10.013915 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:10.013918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:10.013988 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:10.014082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:10.014880 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.015682 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.017065 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.017597 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.017722 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.026704 (+ 8982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:60075: BlockingRecv error: recv got EOF from 127.6.9.193:60075 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:10.499285 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:10.491104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:55837)
0929 11:34:10.491606 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:10.491624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:10.491747 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:10.492030 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:10.492035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:10.492407 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:10.492581 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.492590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.493591 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.493595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.494496 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.494507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.495182 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.495186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.495313 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.495913 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:10.495931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:10.498996 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
W20260929 11:34:10.499497 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:10.491217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:55837 (local address 127.6.9.254:36939)
0929 11:34:10.491445 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:10.491452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:10.491704 (+ 252us) server_negotiation.cc:407] Connection header received
0929 11:34:10.491847 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:10.491851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:10.491914 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:10.491994 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:10.492718 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.493452 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.494626 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.495055 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.495145 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.499210 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:55837: BlockingRecv error: recv got EOF from 127.6.9.194:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:10 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:10.947023 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:10.938187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:35993)
0929 11:34:10.938695 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:10.938712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:10.938854 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:10.939190 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:10.939195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:10.939603 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:10.939759 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.939769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.940958 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.940962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.941972 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:10.941988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.942737 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.942743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:10.942917 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.943585 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:10.943610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:10.946710 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":287,"threads_started":1}
W20260929 11:34:10.947216 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:10.938266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:35993 (local address 127.6.9.254:36939)
0929 11:34:10.938494 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:10.938501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:10.938802 (+ 301us) server_negotiation.cc:407] Connection header received
0929 11:34:10.938969 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:10.938982 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:10.939046 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:10.939132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:10.939917 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.940779 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.942135 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:10.942583 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:10.942701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:10.946964 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:35993: BlockingRecv error: recv got EOF from 127.6.9.195:35993 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:11.036811 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:11.027935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:34871)
0929 11:34:11.028437 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:11.028454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:11.028584 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:11.028981 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:11.028986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:11.029394 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:11.029583 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.029594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.030663 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.030667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.031782 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.031797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.032791 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.032798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.032975 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.033617 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:11.033637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:11.036428 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
W20260929 11:34:11.036963 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:11.028117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:34871 (local address 127.6.9.254:36939)
0929 11:34:11.028359 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:11.028365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:11.028530 (+ 165us) server_negotiation.cc:407] Connection header received
0929 11:34:11.028719 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:11.028725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:11.028792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:11.028876 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:11.029766 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.030515 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.031949 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.032579 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.032737 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.036702 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:34871: BlockingRecv error: recv got EOF from 127.6.9.193:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:11.509184 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:11.500472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:56199)
0929 11:34:11.501051 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:11.501067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:11.501196 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:11.501545 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:11.501550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:11.501958 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:11.502142 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.502155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.503308 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.503314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.504402 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.504417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.505353 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.505358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.505475 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.506094 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:11.506115 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:11.508871 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":317,"threads_started":1}
W20260929 11:34:11.509315 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:11.500763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56199 (local address 127.6.9.254:36939)
0929 11:34:11.500959 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:11.500966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:11.501154 (+ 188us) server_negotiation.cc:407] Connection header received
0929 11:34:11.501296 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:11.501300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:11.501384 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:11.501472 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:11.502282 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.503150 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.504545 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.505186 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.505320 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.509071 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:56199: BlockingRecv error: recv got EOF from 127.6.9.194:56199 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:11 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:11.956068 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:11.948034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:45805)
0929 11:34:11.948541 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:11.948566 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:11.948730 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:11.949051 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:11.949055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:11.949489 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:11.949665 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.949675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.950661 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.950665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.951561 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:11.951573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.952258 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.952263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:11.952430 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.953015 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:11.953033 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:11.955728 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":238,"threads_started":1}
W20260929 11:34:11.956221 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:11.948176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:45805 (local address 127.6.9.254:36939)
0929 11:34:11.948365 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:11.948371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:11.948652 (+ 281us) server_negotiation.cc:407] Connection header received
0929 11:34:11.948857 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:11.948861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:11.948923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:11.948998 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:11.949803 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.950523 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.951697 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:11.952123 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:11.952231 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:11.955948 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:45805: BlockingRecv error: recv got EOF from 127.6.9.195:45805 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:12.046380 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:12.037806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:49683)
0929 11:34:12.038309 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:12.038326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:12.038460 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:12.038817 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:12.038822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:12.039236 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:12.039414 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.039424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.040635 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.040641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.041672 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.041690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.042438 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.042442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.042571 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.043173 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:12.043192 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:12.046091 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":264,"threads_started":1}
W20260929 11:34:12.046578 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:12.037915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:49683 (local address 127.6.9.254:36939)
0929 11:34:12.038219 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:12.038225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:12.038404 (+ 179us) server_negotiation.cc:407] Connection header received
0929 11:34:12.038595 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:12.038601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:12.038682 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:12.038762 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:12.039599 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.040423 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.041850 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.042306 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.042409 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.046342 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:49683: BlockingRecv error: recv got EOF from 127.6.9.193:49683 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:12.518375 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:12.510117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:47741)
0929 11:34:12.510570 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:12.510585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:12.510698 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:12.511089 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:12.511093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:12.511462 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:12.511652 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.511662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.512777 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.512781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.513707 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.513720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.514447 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.514451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.514579 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.515155 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:12.515176 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:12.518030 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
W20260929 11:34:12.518487 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:12.510217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:47741 (local address 127.6.9.254:36939)
0929 11:34:12.510369 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:12.510373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:12.510652 (+ 279us) server_negotiation.cc:407] Connection header received
0929 11:34:12.510836 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:12.510840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:12.510907 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:12.510992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:12.511812 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.512569 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.513864 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.514313 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.514417 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.518287 (+ 3870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:47741: BlockingRecv error: recv got EOF from 127.6.9.194:47741 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:12 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.195@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:12.965471 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:12.957205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.195:48479)
0929 11:34:12.957766 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:12.957784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:12.957918 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:12.958278 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:12.958283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:12.958658 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:12.958833 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.958844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.959861 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.959866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.960847 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:12.960862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.961560 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.961566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:12.961734 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.962335 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:12.962352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:12.965235 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":259,"threads_started":1}
W20260929 11:34:12.965739 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:12.957363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:48479 (local address 127.6.9.254:36939)
0929 11:34:12.957599 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:12.957606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:12.957879 (+ 273us) server_negotiation.cc:407] Connection header received
0929 11:34:12.958066 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:12.958070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:12.958135 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:12.958220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:12.958975 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.959722 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.960987 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:12.961413 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:12.961535 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:12.965505 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.195:48479: BlockingRecv error: recv got EOF from 127.6.9.195:48479 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 11:34:13 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:13 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.193@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:13.056053 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:13.047778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.193:40691)
0929 11:34:13.048271 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:13.048289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:13.048410 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:13.048806 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:13.048810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:13.049232 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:13.049430 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:13.049439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.050539 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.050543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:13.051477 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:13.051488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.052298 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.052303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:13.052430 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.053004 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:13.053020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:13.055751 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":252,"threads_started":1}
W20260929 11:34:13.056185 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:13.047918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:40691 (local address 127.6.9.254:36939)
0929 11:34:13.048115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:13.048120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:13.048377 (+ 257us) server_negotiation.cc:407] Connection header received
0929 11:34:13.048503 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:13.048507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:13.048605 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:13.048722 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:13.049625 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.050420 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.051667 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.052154 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.052263 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.055969 (+ 3706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.193:40691: BlockingRecv error: recv got EOF from 127.6.9.193:40691 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 11:34:13 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:34:13 dist-test-slave-c0l9 krb5kdc[14398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.9.194@KRBTEST.COM for oryx/127.6.9.254@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:34:13.531960 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:13.519673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36939 (local address 127.6.9.194:34717)
0929 11:34:13.520199 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:13.520214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:13.520352 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:13.520734 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:13.520740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:13.521153 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:13.521343 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:13.521352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.522367 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.522371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:13.523309 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:13.523323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.524109 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.524113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:13.524235 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.528460 (+ 4225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:13.528488 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:13.531589 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.9.254:36939: Server oryx/127.6.9.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
W20260929 11:34:13.532068 15135 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:13.519749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34717 (local address 127.6.9.254:36939)
0929 11:34:13.519926 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:13.519932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:13.520301 (+ 369us) server_negotiation.cc:407] Connection header received
0929 11:34:13.520471 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:13.520475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:13.520552 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:13.520647 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:13.521475 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.522211 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.523472 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.523952 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.524062 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.531835 (+ 7773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.9.194:34717: BlockingRecv error: recv got EOF from 127.6.9.194:34717 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:34:13.807335 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14985
I20260929 11:34:13.820356 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:13.808629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.6.9.193:44453)
0929 11:34:13.809244 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:13.809252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:13.809280 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:34:13.809348 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:13.809356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:13.809570 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:13.809742 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:13.811201 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.812360 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.813835 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.814443 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.814556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.817136 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:13.817150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:13.817170 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:13.817248 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:13.819791 (+ 2543us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:34:13.820095 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58832: Request ticket server kudu/127.6.9.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
W20260929 11:34:13.820469 15446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58832: Request ticket server kudu/127.6.9.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:34:13.820612 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14985: Not authorized: Client connection negotiation failed: client connection to 127.6.9.193:44453: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:34:13.820861 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14985
I20260929 11:34:13.834163 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15137
I20260929 11:34:13.846239 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:13.835163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.6.9.194:36845)
0929 11:34:13.835758 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:13.835765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:13.835796 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:34:13.835861 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:13.835867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:13.836078 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:13.836247 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:13.837504 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.838566 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.839861 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.840558 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.840741 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.843199 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:13.843208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:13.843223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:13.843276 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:13.845730 (+ 2454us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:34:13.846002 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60972: Request ticket server kudu/127.6.9.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":276,"threads_started":1}
W20260929 11:34:13.846342 15447 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60972: Request ticket server kudu/127.6.9.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:34:13.846482 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15137: Not authorized: Client connection negotiation failed: client connection to 127.6.9.194:36845: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:34:13.846684 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15137
I20260929 11:34:13.860201 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15268
I20260929 11:34:13.872610 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:13.861324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.6.9.195:44441)
0929 11:34:13.861893 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:13.861901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:13.861926 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:34:13.862000 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:13.862006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:13.862230 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:13.862386 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:13.863743 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.864901 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.866157 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:13.866743 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:13.866842 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:13.869484 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:13.869494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:13.869509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:13.869561 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:13.872060 (+ 2499us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:34:13.872379 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55056: Request ticket server kudu/127.6.9.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":292,"threads_started":1}
W20260929 11:34:13.872743 15448 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55056: Request ticket server kudu/127.6.9.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:34:13.872893 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15268: Not authorized: Client connection negotiation failed: client connection to 127.6.9.195:44441: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.9.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:34:13.873085 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15268
I20260929 11:34:13.886355 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 14916
I20260929 11:34:13.887719 14978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:14.002351 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 14916
2026-09-29T11:34:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20964 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T11:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:34:14Z Disabled control of system clock
I20260929 11:34:14.073156 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33639
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:34:14.402076 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:14.402395 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:14.402494 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:14.409884 15453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:34:14.410024 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:14.410063 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:14.410152 15453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:34:14.410208 15453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:34:14.419782 15453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:14.421252 15453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:14.423120 15453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:14.436318 15458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:14.436316 15459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:14.436318 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:14.436604 15453 server_base.cc:1034] running on GCE node
I20260929 11:34:14.437701 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:14.439283 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:14.440549 15453 hybrid_clock.cc:648] HybridClock initialized: now 1790681654440503 us; error 33 us; skew 500 ppm
I20260929 11:34:14.440985 15453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:14.443423 15453 webserver.cc:460] Webserver started at http://127.6.9.254:38645/ using document root <none> and password file <none>
I20260929 11:34:14.444139 15453 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:14.444244 15453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:14.444563 15453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:14.448269 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "502335765a854817b94283db15d80c2b"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "42707ca5f68b3d6e75869f4f3cb32937"
server_key_iv: "7d439c4f4795eace62204f0959ea842b"
server_key_version: "encryptionkey@0"
I20260929 11:34:14.449462 15453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "502335765a854817b94283db15d80c2b"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "42707ca5f68b3d6e75869f4f3cb32937"
server_key_iv: "7d439c4f4795eace62204f0959ea842b"
server_key_version: "encryptionkey@0"
I20260929 11:34:14.455324 15453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:34:14.459200 15467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:14.460901 15453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:34:14.461118 15453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "502335765a854817b94283db15d80c2b"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "42707ca5f68b3d6e75869f4f3cb32937"
server_key_iv: "7d439c4f4795eace62204f0959ea842b"
server_key_version: "encryptionkey@0"
I20260929 11:34:14.461366 15453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:14.483789 15453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:14.484784 15453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:14.485052 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:14.502564 15453 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33639
I20260929 11:34:14.502583 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33639 every 8 connection(s)
I20260929 11:34:14.503942 15453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:34:14.504813 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15453
I20260929 11:34:14.505012 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:34:14.505709 6183 external_mini_cluster.cc:1428] Setting key 685a568fdca117445facb5651699031d
I20260929 11:34:14.508303 15519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:14.519665 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:14.507720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.6.9.254:33639)
0929 11:34:14.508697 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:14.508714 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:14.508758 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:34:14.510347 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:14.510368 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:14.510838 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:14.511192 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:14.511844 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:14.513257 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:14.515149 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:14.515838 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:14.515982 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:14.517318 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:14.517326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:14.517341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:14.517921 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:14.517998 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:14.518022 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:14.518138 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:14.518355 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 11:34:14.518703 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
I20260929 11:34:14.520531 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b: Bootstrap starting.
I20260929 11:34:14.523996 15519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:14.525007 15519 log.cc:824] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:14.527930 15519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b: No bootstrap required, opened a new log
I20260929 11:34:14.535394 15519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } }
I20260929 11:34:14.535780 15519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:14.535883 15519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502335765a854817b94283db15d80c2b, State: Initialized, Role: FOLLOWER
I20260929 11:34:14.536422 15519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } }
I20260929 11:34:14.536635 15519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:34:14.536784 15519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:34:14.536922 15519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:14.539033 15519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } }
I20260929 11:34:14.539512 15519 leader_election.cc:302] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502335765a854817b94283db15d80c2b; no voters:
I20260929 11:34:14.540165 15519 leader_election.cc:288] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:34:14.540342 15524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:34:14.542261 15519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:34:14.542347 15524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [term 1 LEADER]: Becoming Leader. State: Replica: 502335765a854817b94283db15d80c2b, State: Running, Role: LEADER
I20260929 11:34:14.542878 15524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } }
I20260929 11:34:14.545810 15526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502335765a854817b94283db15d80c2b. Latest consensus state: current_term: 1 leader_uuid: "502335765a854817b94283db15d80c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } } }
I20260929 11:34:14.545768 15525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502335765a854817b94283db15d80c2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502335765a854817b94283db15d80c2b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33639 } } }
I20260929 11:34:14.546264 15526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:14.546273 15525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:14.548554 15533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:34:14.552625 15533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:34:14.561597 15533 catalog_manager.cc:1347] Generated new cluster ID: 025cb508662e436193c0c47e0d95e288
I20260929 11:34:14.561712 15533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:34:14.592986 15533 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:34:14.594271 15533 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:34:14.600503 15533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502335765a854817b94283db15d80c2b: Generated new TSK 0
I20260929 11:34:14.601661 15533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:34:14.617882 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--builtin_ntp_servers=127.6.9.212:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 11:34:14.951885 15543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:14.952196 15543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:14.952286 15543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:14.959602 15543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:14.959728 15543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:14.959841 15543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:34:14.970376 15543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:14.971678 15543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:14.973543 15543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:14.988854 15548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:14.988794 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:14.988832 15549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:14.989796 15543 server_base.cc:1034] running on GCE node
I20260929 11:34:14.990483 15543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:14.991528 15543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:14.992743 15543 hybrid_clock.cc:648] HybridClock initialized: now 1790681654992680 us; error 47 us; skew 500 ppm
I20260929 11:34:14.993072 15543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:14.995456 15543 webserver.cc:460] Webserver started at http://127.6.9.193:39221/ using document root <none> and password file <none>
I20260929 11:34:14.996187 15543 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:14.996290 15543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:14.996614 15543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:14.999528 15543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "a0247eaa0286498aaeb87106f7f2743d"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "7510d5e637a628dbfc7672eb4f926952"
server_key_iv: "1707f333f27d9395e1f91c7c345b5615"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.000375 15543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "a0247eaa0286498aaeb87106f7f2743d"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "7510d5e637a628dbfc7672eb4f926952"
server_key_iv: "1707f333f27d9395e1f91c7c345b5615"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.005841 15543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:34:15.009413 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.010912 15543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:34:15.011117 15543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "a0247eaa0286498aaeb87106f7f2743d"
format_stamp: "Formatted at 2026-09-29 11:34:14 on dist-test-slave-c0l9"
server_key: "7510d5e637a628dbfc7672eb4f926952"
server_key_iv: "1707f333f27d9395e1f91c7c345b5615"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.011355 15543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:15.052815 15543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:15.053728 15543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:15.054003 15543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:15.054980 15543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:15.056567 15543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:15.056702 15543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.056821 15543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:15.056893 15543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.088295 15543 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:46525
I20260929 11:34:15.088338 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:46525 every 8 connection(s)
I20260929 11:34:15.089598 15543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:34:15.093164 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15543
I20260929 11:34:15.093366 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:34:15.094163 6183 external_mini_cluster.cc:1428] Setting key 5f3affcc1d8c02f1d65c58c165b84378
I20260929 11:34:15.097126 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--builtin_ntp_servers=127.6.9.212:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:34:15.103473 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:15.092156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44441 (local address 127.6.9.254:33639)
0929 11:34:15.092587 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:15.092592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:15.094001 (+ 1409us) server_negotiation.cc:407] Connection header received
0929 11:34:15.096168 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:15.096173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:15.096251 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:15.096342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:15.097571 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.098585 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.100335 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.100933 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.101065 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:15.102468 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:15.102474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:15.102479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:15.102503 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:15.102552 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:15.102567 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:15.102659 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:15.102987 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 11:34:15.103105 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
I20260929 11:34:15.104687 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:15.092648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33639 (local address 127.6.9.193:44441)
0929 11:34:15.093785 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:15.093843 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:15.095888 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:15.096588 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:15.096598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:15.096733 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:15.097380 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:15.097399 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.098778 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.098782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:15.100171 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:15.100184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.101179 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.101184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:15.101417 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:15.102129 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:15.102132 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:15.102176 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:15.102267 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:15.102697 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:15.102708 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:15.102716 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:34:15.102958 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 11:34:15.103333 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":316,"threads_started":1}
I20260929 11:34:15.106374 15670 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33639
I20260929 11:34:15.106684 15670 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:15.107458 15670 heartbeater.cc:511] Master 127.6.9.254:33639 requested a full tablet report, sending...
I20260929 11:34:15.109799 15484 ts_manager.cc:194] Registered new tserver with Master: a0247eaa0286498aaeb87106f7f2743d (127.6.9.193:46525)
I20260929 11:34:15.111914 15484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.193:44441
W20260929 11:34:15.422384 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:15.422695 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:15.422780 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:15.430155 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:15.430272 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:15.430378 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:34:15.442688 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:15.444041 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:15.445942 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:15.460426 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:15.460459 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:15.460614 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:15.461171 15674 server_base.cc:1034] running on GCE node
I20260929 11:34:15.461876 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:15.462886 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:15.464110 15674 hybrid_clock.cc:648] HybridClock initialized: now 1790681655464033 us; error 75 us; skew 500 ppm
I20260929 11:34:15.464459 15674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:15.466928 15674 webserver.cc:460] Webserver started at http://127.6.9.194:46613/ using document root <none> and password file <none>
I20260929 11:34:15.467834 15674 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:15.467967 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:15.468318 15674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:15.471417 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "cd42a3658bac4eb289ae5cd53cdc7b47"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "4dc9ea4f9a1ba4d369ce30c34d3fef5f"
server_key_iv: "f21898c7aad50f95b63db7e6f41472b5"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.472301 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "cd42a3658bac4eb289ae5cd53cdc7b47"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "4dc9ea4f9a1ba4d369ce30c34d3fef5f"
server_key_iv: "f21898c7aad50f95b63db7e6f41472b5"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.477828 15674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:34:15.481690 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.483168 15674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 11:34:15.483366 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "cd42a3658bac4eb289ae5cd53cdc7b47"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "4dc9ea4f9a1ba4d369ce30c34d3fef5f"
server_key_iv: "f21898c7aad50f95b63db7e6f41472b5"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.483592 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:15.513752 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:15.514693 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:15.514986 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:15.516026 15674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:15.517650 15674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:15.517750 15674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.517859 15674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:15.517926 15674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.547720 15674 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:39709
I20260929 11:34:15.547788 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:39709 every 8 connection(s)
I20260929 11:34:15.549029 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:34:15.551695 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15674
I20260929 11:34:15.551854 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:34:15.552434 6183 external_mini_cluster.cc:1428] Setting key 67e3c065b0318ef943e41ae96715c575
I20260929 11:34:15.555410 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--builtin_ntp_servers=127.6.9.212:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:34:15.561761 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:15.551375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:44601 (local address 127.6.9.254:33639)
0929 11:34:15.551528 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:15.551532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:15.553586 (+ 2054us) server_negotiation.cc:407] Connection header received
0929 11:34:15.555196 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:15.555203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:15.555287 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:15.555403 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:15.556473 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.557460 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.559000 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.559551 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.559672 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:15.561020 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:15.561027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:15.561031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:15.561057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:15.561093 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:15.561117 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:15.561194 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:15.561397 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 11:34:15.561505 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 11:34:15.563277 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:15.552079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33639 (local address 127.6.9.194:44601)
0929 11:34:15.553139 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:15.553187 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:15.554896 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:15.555604 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:15.555616 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:15.555734 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:15.556301 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:15.556319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.557609 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.557613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:15.558860 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:15.558873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:15.559772 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:15.559778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:15.559986 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:15.560694 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:15.560698 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:15.560750 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:15.560835 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:15.561204 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:15.561211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:15.561216 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:34:15.561380 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:34:15.561941 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":328,"threads_started":1}
I20260929 11:34:15.564749 15801 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33639
I20260929 11:34:15.565088 15801 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:15.565966 15801 heartbeater.cc:511] Master 127.6.9.254:33639 requested a full tablet report, sending...
I20260929 11:34:15.567584 15484 ts_manager.cc:194] Registered new tserver with Master: cd42a3658bac4eb289ae5cd53cdc7b47 (127.6.9.194:39709)
I20260929 11:34:15.568436 15484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.194:44601
W20260929 11:34:15.889729 15804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:15.890018 15804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:15.890137 15804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:15.897578 15804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:15.897698 15804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:15.897837 15804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:34:15.907649 15804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:15.909037 15804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:15.910804 15804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:15.923338 15809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:15.923525 15810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:15.923682 15812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:15.924340 15804 server_base.cc:1034] running on GCE node
I20260929 11:34:15.924993 15804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:15.925951 15804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:15.927156 15804 hybrid_clock.cc:648] HybridClock initialized: now 1790681655927093 us; error 52 us; skew 500 ppm
I20260929 11:34:15.927470 15804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:15.929844 15804 webserver.cc:460] Webserver started at http://127.6.9.195:41561/ using document root <none> and password file <none>
I20260929 11:34:15.930544 15804 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:15.930648 15804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:15.930953 15804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:15.933809 15804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "6b7bcbd0ee9b40c1b9a59f780f034690"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "b64fdd7371eea40f8f4fddb050900b9d"
server_key_iv: "59647c9ff65eb094350d3ae5ac1638b2"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.934592 15804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "6b7bcbd0ee9b40c1b9a59f780f034690"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "b64fdd7371eea40f8f4fddb050900b9d"
server_key_iv: "59647c9ff65eb094350d3ae5ac1638b2"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.940070 15804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 11:34:15.943964 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.945523 15804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:15.945691 15804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "6b7bcbd0ee9b40c1b9a59f780f034690"
format_stamp: "Formatted at 2026-09-29 11:34:15 on dist-test-slave-c0l9"
server_key: "b64fdd7371eea40f8f4fddb050900b9d"
server_key_iv: "59647c9ff65eb094350d3ae5ac1638b2"
server_key_version: "encryptionkey@0"
I20260929 11:34:15.945909 15804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:15.990664 15804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:15.991597 15804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:15.991838 15804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:15.992830 15804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:15.994383 15804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:15.994477 15804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:15.994565 15804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:15.994617 15804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:16.026147 15804 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:38285
I20260929 11:34:16.026201 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:38285 every 8 connection(s)
I20260929 11:34:16.027448 15804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:34:16.031913 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15804
I20260929 11:34:16.032109 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:34:16.032572 6183 external_mini_cluster.cc:1428] Setting key 9c65f7595bc48e25a565f79a7aba21b7
I20260929 11:34:16.040875 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.030234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:52343 (local address 127.6.9.254:33639)
0929 11:34:16.030416 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.030421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.032037 (+ 1616us) server_negotiation.cc:407] Connection header received
0929 11:34:16.034086 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.034092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.034189 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.034303 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.035487 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.036337 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.038035 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.038463 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.038584 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:16.039974 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:16.039979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:16.039982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:16.040003 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:16.040035 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:16.040047 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:16.040121 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:16.040381 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 11:34:16.040491 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:34:16.041996 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.030749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33639 (local address 127.6.9.195:52343)
0929 11:34:16.031847 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:16.031887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:16.033769 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:16.034510 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:16.034521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:16.034656 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:16.035301 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:16.035322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.036517 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.036521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:16.037867 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:16.037882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.038707 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.038715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:16.038946 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:16.039626 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:16.039628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:16.039677 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:16.039768 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:16.040136 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:16.040149 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:16.040156 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:34:16.040343 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 11:34:16.040817 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":333,"threads_started":1}
I20260929 11:34:16.043618 15931 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33639
I20260929 11:34:16.043959 15931 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:16.044859 15931 heartbeater.cc:511] Master 127.6.9.254:33639 requested a full tablet report, sending...
I20260929 11:34:16.046576 15484 ts_manager.cc:194] Registered new tserver with Master: 6b7bcbd0ee9b40c1b9a59f780f034690 (127.6.9.195:38285)
I20260929 11:34:16.047605 15484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.195:52343
I20260929 11:34:16.048609 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:34:16.054924 15672 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:16.053336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.6.9.254:33639)
0929 11:34:16.053599 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.053605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.053862 (+ 257us) server_negotiation.cc:407] Connection header received
0929 11:34:16.054053 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.054058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.054140 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.054242 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.054687 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47240: BlockingRecv error: recv got EOF from 127.0.0.1:47240 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:34:16.055573 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15543
I20260929 11:34:16.063213 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.056508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.6.9.193:46525)
0929 11:34:16.057033 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.057040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.057064 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:34:16.057115 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.057121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.057329 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.057464 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.057873 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.058841 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.060055 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.060623 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.060783 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:16.061797 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:16.061806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:16.061820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:16.062414 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:16.062460 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:16.062480 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:16.062601 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:16.062756 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 11:34:16.062885 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260929 11:34:16.064421 15665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:16.144870 15670 heartbeater.cc:503] Master 127.6.9.254:33639 was elected leader, sending a full tablet report...
I20260929 11:34:16.146240 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15543
W20260929 11:34:16.159507 15470 connection.cc:535] server connection from 127.6.9.193:44441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:34:16.159790 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15674
I20260929 11:34:16.167441 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.160695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.6.9.194:39709)
0929 11:34:16.161209 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.161216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.161243 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:34:16.161310 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.161323 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.161547 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.161677 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.162028 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.163158 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.164441 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.165081 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.165201 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:16.166179 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:16.166187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:16.166201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:16.166766 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:16.166811 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:16.166843 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:16.166926 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:16.167026 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 11:34:16.167153 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260929 11:34:16.168612 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:16.248870 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15674
I20260929 11:34:16.261904 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15804
I20260929 11:34:16.270303 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.262869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.6.9.195:38285)
0929 11:34:16.263518 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.263525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.263555 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:34:16.263612 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.263618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.263832 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.263986 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.264418 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.265581 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.266930 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:16.267616 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:16.267763 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:16.268844 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:16.268853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:16.268869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:16.269467 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:16.269520 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:16.269540 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:16.269653 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:16.269815 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 11:34:16.269957 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":344,"threads_started":1}
I20260929 11:34:16.271611 15926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:16.352815 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15804
I20260929 11:34:16.365300 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15453
I20260929 11:34:16.366614 15514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:16.469476 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15453
2026-09-29T11:34:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2462 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T11:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:34:16Z Disabled control of system clock
I20260929 11:34:16.535409 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40173
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:34:16.867367 15950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:16.867686 15950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:16.867770 15950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:16.875281 15950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:34:16.875403 15950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:16.875439 15950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:16.875499 15950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:34:16.875542 15950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:34:16.885111 15950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:16.886435 15950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:16.888231 15950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:16.899947 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:16.900069 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:16.900543 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:16.900770 15950 server_base.cc:1034] running on GCE node
I20260929 11:34:16.901611 15950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:16.903074 15950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:16.904320 15950 hybrid_clock.cc:648] HybridClock initialized: now 1790681656904253 us; error 64 us; skew 500 ppm
I20260929 11:34:16.904708 15950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:16.907199 15950 webserver.cc:460] Webserver started at http://127.6.9.254:37111/ using document root <none> and password file <none>
I20260929 11:34:16.907919 15950 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:16.908011 15950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:16.908299 15950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:16.911316 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "8683698e60774433bd6941181559e591"
format_stamp: "Formatted at 2026-09-29 11:34:16 on dist-test-slave-c0l9"
server_key: "99e01b28856e952aac0898e5c36ebab8"
server_key_iv: "eeef94c8a8bf5b499e8195d69c1e8a5f"
server_key_version: "encryptionkey@0"
I20260929 11:34:16.912160 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "8683698e60774433bd6941181559e591"
format_stamp: "Formatted at 2026-09-29 11:34:16 on dist-test-slave-c0l9"
server_key: "99e01b28856e952aac0898e5c36ebab8"
server_key_iv: "eeef94c8a8bf5b499e8195d69c1e8a5f"
server_key_version: "encryptionkey@0"
I20260929 11:34:16.917819 15950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:34:16.921671 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:16.923182 15950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:34:16.923384 15950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "8683698e60774433bd6941181559e591"
format_stamp: "Formatted at 2026-09-29 11:34:16 on dist-test-slave-c0l9"
server_key: "99e01b28856e952aac0898e5c36ebab8"
server_key_iv: "eeef94c8a8bf5b499e8195d69c1e8a5f"
server_key_version: "encryptionkey@0"
I20260929 11:34:16.923597 15950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:16.929793 15950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:16.930675 15950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:16.930949 15950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:16.950446 15950 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40173
I20260929 11:34:16.950488 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40173 every 8 connection(s)
I20260929 11:34:16.951843 15950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:34:16.955971 16016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:16.957090 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 15950
I20260929 11:34:16.957334 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:34:16.958202 6183 external_mini_cluster.cc:1428] Setting key b3ca3102af44bf008622b2cfe9449092
I20260929 11:34:16.967332 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:16.960703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.6.9.254:40173)
0929 11:34:16.961871 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:16.961883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:16.961932 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:34:16.963555 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:16.963583 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:16.964134 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:16.964598 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:16.965706 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:16.965715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:16.965729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:16.965744 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:16.965807 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:16.965823 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:16.966030 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:16.966151 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 11:34:16.966408 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":445,"threads_started":1}
I20260929 11:34:16.970384 16016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591: Bootstrap starting.
I20260929 11:34:16.973690 16016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:16.974828 16016 log.cc:824] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:16.977801 16016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591: No bootstrap required, opened a new log
I20260929 11:34:16.985553 16016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } }
I20260929 11:34:16.985976 16016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:16.986099 16016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8683698e60774433bd6941181559e591, State: Initialized, Role: FOLLOWER
I20260929 11:34:16.986706 16016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } }
I20260929 11:34:16.986943 16016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:34:16.987082 16016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:34:16.987250 16016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:16.989611 16016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } }
I20260929 11:34:16.990185 16016 leader_election.cc:302] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8683698e60774433bd6941181559e591; no voters:
I20260929 11:34:16.990804 16016 leader_election.cc:288] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:34:16.990923 16021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:34:16.991616 16021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [term 1 LEADER]: Becoming Leader. State: Replica: 8683698e60774433bd6941181559e591, State: Running, Role: LEADER
I20260929 11:34:16.992106 16021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } }
I20260929 11:34:16.993036 16016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:34:16.994992 16023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8683698e60774433bd6941181559e591. Latest consensus state: current_term: 1 leader_uuid: "8683698e60774433bd6941181559e591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } } }
I20260929 11:34:16.995370 16023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:16.996522 16022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8683698e60774433bd6941181559e591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8683698e60774433bd6941181559e591" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40173 } } }
I20260929 11:34:16.996887 16022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591 [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:16.998752 16028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:34:17.004103 16028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:34:17.014614 16028 catalog_manager.cc:1347] Generated new cluster ID: fe575bfc5db14a7fa7e673fe92947e5e
I20260929 11:34:17.014766 16028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:34:17.050110 16028 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:34:17.050520 16028 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:34:17.059139 16028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8683698e60774433bd6941181559e591: Generated new TSK 0
I20260929 11:34:17.060055 16028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:34:17.075732 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--builtin_ntp_servers=127.6.9.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:34:17.426589 16040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:17.426892 16040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:17.426982 16040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:17.435245 16040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:17.435395 16040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:17.435638 16040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:34:17.447350 16040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:17.448647 16040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:17.450600 16040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:17.465181 16045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:17.465322 16048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:17.465126 16046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:17.465905 16040 server_base.cc:1034] running on GCE node
I20260929 11:34:17.466586 16040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:17.467741 16040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:17.468956 16040 hybrid_clock.cc:648] HybridClock initialized: now 1790681657468882 us; error 70 us; skew 500 ppm
I20260929 11:34:17.469300 16040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:17.471698 16040 webserver.cc:460] Webserver started at http://127.6.9.193:45931/ using document root <none> and password file <none>
I20260929 11:34:17.472478 16040 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:17.472591 16040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:17.472960 16040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:17.477604 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "e6c20e8db38c44d3adea504d0088b8d0"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "9c1e61c019a6e5ef4a640593e532c0ff"
server_key_iv: "9a6cab1dae989de15dfba78c5050e978"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.478487 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "e6c20e8db38c44d3adea504d0088b8d0"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "9c1e61c019a6e5ef4a640593e532c0ff"
server_key_iv: "9a6cab1dae989de15dfba78c5050e978"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.484148 16040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:34:17.487865 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.489396 16040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:17.489576 16040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "e6c20e8db38c44d3adea504d0088b8d0"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "9c1e61c019a6e5ef4a640593e532c0ff"
server_key_iv: "9a6cab1dae989de15dfba78c5050e978"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.489801 16040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:17.495744 16040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:17.496619 16040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:17.496888 16040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:17.497841 16040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:17.499336 16040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:17.499441 16040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.499549 16040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:17.499621 16040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.529729 16040 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:33583
I20260929 11:34:17.529788 16166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:33583 every 8 connection(s)
I20260929 11:34:17.531117 16040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:34:17.539558 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:17.533796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:45389 (local address 127.6.9.254:40173)
0929 11:34:17.534216 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:17.534221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:17.535396 (+ 1175us) server_negotiation.cc:407] Connection header received
0929 11:34:17.537158 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:17.537164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:17.537231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:17.537337 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:17.538826 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:17.538833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:17.538839 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:17.538863 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:17.538919 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:17.538938 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:17.539079 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:17.539270 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 11:34:17.539328 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":203,"threads_started":1}
I20260929 11:34:17.540512 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:17.534331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40173 (local address 127.6.9.193:45389)
0929 11:34:17.535227 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:17.535273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:17.536874 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:17.537575 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:17.537584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:17.537682 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:17.538405 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:17.538408 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:17.538462 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:17.538577 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:17.539054 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:17.539065 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:17.539075 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:34:17.539328 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 11:34:17.539550 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260929 11:34:17.541597 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16040
I20260929 11:34:17.541795 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:34:17.542398 16167 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40173
I20260929 11:34:17.542548 6183 external_mini_cluster.cc:1428] Setting key b6344bea338ccfc5604e2fb9cf18ead5
I20260929 11:34:17.542737 16167 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:17.543485 16167 heartbeater.cc:511] Master 127.6.9.254:40173 requested a full tablet report, sending...
I20260929 11:34:17.545544 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--builtin_ntp_servers=127.6.9.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:34:17.545730 15981 ts_manager.cc:194] Registered new tserver with Master: e6c20e8db38c44d3adea504d0088b8d0 (127.6.9.193:33583)
W20260929 11:34:17.867461 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:17.867775 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:17.867865 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:17.875360 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:17.875486 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:17.875604 16171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:34:17.884928 16171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:17.886267 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:17.888056 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:17.901461 16176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:17.901429 16177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:17.901397 16179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:17.902127 16171 server_base.cc:1034] running on GCE node
I20260929 11:34:17.902794 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:17.903801 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:17.905005 16171 hybrid_clock.cc:648] HybridClock initialized: now 1790681657904949 us; error 56 us; skew 500 ppm
I20260929 11:34:17.905346 16171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:17.907850 16171 webserver.cc:460] Webserver started at http://127.6.9.194:41989/ using document root <none> and password file <none>
I20260929 11:34:17.908620 16171 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:17.908778 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:17.909111 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:17.912055 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "b99b7ce4764f4d329a042b32146e9bbd"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "5c5b19c5ffdd90e02761fb25d52de4fd"
server_key_iv: "d40192f5e96533bedd5ea01c6fbf6f44"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.912878 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "b99b7ce4764f4d329a042b32146e9bbd"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "5c5b19c5ffdd90e02761fb25d52de4fd"
server_key_iv: "d40192f5e96533bedd5ea01c6fbf6f44"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.918243 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:34:17.921908 16185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.923328 16171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:17.923511 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "b99b7ce4764f4d329a042b32146e9bbd"
format_stamp: "Formatted at 2026-09-29 11:34:17 on dist-test-slave-c0l9"
server_key: "5c5b19c5ffdd90e02761fb25d52de4fd"
server_key_iv: "d40192f5e96533bedd5ea01c6fbf6f44"
server_key_version: "encryptionkey@0"
I20260929 11:34:17.923753 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:17.929903 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:17.930804 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:17.931068 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:17.932056 16171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:17.933579 16171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:17.933686 16171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.933794 16171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:17.933866 16171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:17.965016 16171 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:44355
I20260929 11:34:17.965054 16297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:44355 every 8 connection(s)
I20260929 11:34:17.966462 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:34:17.967047 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16171
I20260929 11:34:17.967263 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:34:17.967847 6183 external_mini_cluster.cc:1428] Setting key 767133efd5f7baca0d4bd10fff07ced7
I20260929 11:34:17.970989 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--builtin_ntp_servers=127.6.9.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:34:17.975320 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:17.969361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:41417 (local address 127.6.9.254:40173)
0929 11:34:17.969560 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:17.969567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:17.971084 (+ 1517us) server_negotiation.cc:407] Connection header received
0929 11:34:17.973025 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:17.973031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:17.973099 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:17.973196 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:17.974561 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:17.974567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:17.974571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:17.974592 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:17.974626 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:17.974638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:17.974745 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:17.974934 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 11:34:17.975007 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:34:17.976317 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:17.969930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40173 (local address 127.6.9.194:41417)
0929 11:34:17.970926 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:17.970963 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:17.972747 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:17.973450 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:17.973458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:17.973563 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:17.974227 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:17.974230 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:17.974270 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:17.974369 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:17.974728 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:17.974738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:17.974746 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:34:17.975013 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 11:34:17.975301 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":326,"threads_started":1}
I20260929 11:34:17.977948 16298 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40173
I20260929 11:34:17.978283 16298 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:17.978987 16298 heartbeater.cc:511] Master 127.6.9.254:40173 requested a full tablet report, sending...
I20260929 11:34:17.980753 15981 ts_manager.cc:194] Registered new tserver with Master: b99b7ce4764f4d329a042b32146e9bbd (127.6.9.194:44355)
W20260929 11:34:18.289816 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:18.290108 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:18.290194 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:18.297632 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:18.297739 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:18.297848 16301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:34:18.307075 16301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:18.308338 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:18.310060 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:18.322659 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:18.322870 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:18.322800 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:18.323772 16301 server_base.cc:1034] running on GCE node
I20260929 11:34:18.324405 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:18.325408 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:18.326653 16301 hybrid_clock.cc:648] HybridClock initialized: now 1790681658326587 us; error 42 us; skew 500 ppm
I20260929 11:34:18.327014 16301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:34:18.329546 16301 webserver.cc:460] Webserver started at http://127.6.9.195:35371/ using document root <none> and password file <none>
I20260929 11:34:18.330253 16301 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:18.330348 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:18.330686 16301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:18.333899 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "b4b9f507946645bc96e157739f1d42cf"
format_stamp: "Formatted at 2026-09-29 11:34:18 on dist-test-slave-c0l9"
server_key: "aa197f0a2324891cdbd166fa8f42895b"
server_key_iv: "4dd78260f28a870e8f542c09cb91b42b"
server_key_version: "encryptionkey@0"
I20260929 11:34:18.334719 16301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "b4b9f507946645bc96e157739f1d42cf"
format_stamp: "Formatted at 2026-09-29 11:34:18 on dist-test-slave-c0l9"
server_key: "aa197f0a2324891cdbd166fa8f42895b"
server_key_iv: "4dd78260f28a870e8f542c09cb91b42b"
server_key_version: "encryptionkey@0"
I20260929 11:34:18.339995 16301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:34:18.343577 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:18.344955 16301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:18.345120 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b4b9f507946645bc96e157739f1d42cf"
format_stamp: "Formatted at 2026-09-29 11:34:18 on dist-test-slave-c0l9"
server_key: "aa197f0a2324891cdbd166fa8f42895b"
server_key_iv: "4dd78260f28a870e8f542c09cb91b42b"
server_key_version: "encryptionkey@0"
I20260929 11:34:18.345335 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:18.351032 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:18.351837 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:18.352083 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:18.353044 16301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:18.354508 16301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:18.354602 16301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:18.354712 16301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:18.354786 16301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:18.384648 16301 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:33509
I20260929 11:34:18.384758 16427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:33509 every 8 connection(s)
I20260929 11:34:18.386003 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:34:18.394181 16169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:18.388607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54403 (local address 127.6.9.254:40173)
0929 11:34:18.388799 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:18.388803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:18.390254 (+ 1451us) server_negotiation.cc:407] Connection header received
0929 11:34:18.391873 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:18.391878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:18.391938 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:18.392032 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:18.393461 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:18.393467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:18.393470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:18.393490 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:18.393525 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:18.393537 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:18.393635 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:18.393869 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 11:34:18.393924 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:34:18.394620 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16301
I20260929 11:34:18.394801 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:34:18.395254 6183 external_mini_cluster.cc:1428] Setting key 80335520090ea336f1fb4cd0a568a371
I20260929 11:34:18.395331 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:18.389125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40173 (local address 127.6.9.195:54403)
0929 11:34:18.390053 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:18.390093 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:18.391591 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:18.392307 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:18.392318 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:18.392418 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:18.393127 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:34:18.393130 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:18.393171 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:34:18.393265 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:18.393661 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:18.393673 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:18.393683 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:34:18.393930 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 11:34:18.394197 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
I20260929 11:34:18.396909 16428 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40173
I20260929 11:34:18.397286 16428 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:18.397918 16428 heartbeater.cc:511] Master 127.6.9.254:40173 requested a full tablet report, sending...
I20260929 11:34:18.399500 15980 ts_manager.cc:194] Registered new tserver with Master: b4b9f507946645bc96e157739f1d42cf (127.6.9.195:33509)
I20260929 11:34:18.411206 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:34:18.417520 16169 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:34:18.415791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.6.9.254:40173)
0929 11:34:18.416111 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:18.416117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:18.416412 (+ 295us) server_negotiation.cc:407] Connection header received
0929 11:34:18.416557 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:18.416561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:18.416635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:18.416753 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:18.417296 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46914: BlockingRecv error: recv got EOF from 127.0.0.1:46914 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 11:34:18.417912 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16040
I20260929 11:34:18.421813 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:18.419080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.6.9.193:33583)
0929 11:34:18.419623 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:18.419630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:18.419655 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:34:18.419716 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:18.419722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:18.419933 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:18.420074 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:18.421082 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:18.421088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:18.421099 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:18.421111 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:18.421156 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:18.421179 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:18.421297 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:18.421420 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 11:34:18.421536 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
I20260929 11:34:18.423019 16162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:18.507081 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16040
I20260929 11:34:18.520473 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16171
I20260929 11:34:18.524086 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:18.521467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.6.9.194:44355)
0929 11:34:18.521961 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:18.521976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:18.522003 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:34:18.522066 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:18.522071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:18.522300 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:18.522444 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:18.523391 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:18.523396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:18.523408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:18.523426 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:18.523470 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:18.523484 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:18.523602 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:18.523731 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 11:34:18.523832 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":248,"threads_started":1}
I20260929 11:34:18.525352 16293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:18.605985 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16171
I20260929 11:34:18.618639 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16301
I20260929 11:34:18.629169 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:18.619603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.6.9.195:33509)
0929 11:34:18.620248 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:18.620255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:18.620287 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:34:18.620366 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:18.620374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:18.620603 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:18.620776 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:18.628267 (+ 7491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:18.628278 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:18.628293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:34:18.628326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:18.628390 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:34:18.628410 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:18.628613 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:18.628708 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 11:34:18.628827 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
I20260929 11:34:18.630484 16423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:18.712167 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16301
I20260929 11:34:18.724751 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 15950
I20260929 11:34:18.725903 16011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:18.828056 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 15950
2026-09-29T11:34:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2360 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:34:18 dist-test-slave-c0l9 krb5kdc[16446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:34:18 dist-test-slave-c0l9 krb5kdc[16446](info): set up 2 sockets
Sep 29 11:34:18 dist-test-slave-c0l9 krb5kdc[16446](info): commencing operation
krb5kdc: starting...
W20260929 11:34:19.833324 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.007s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:34:19 dist-test-slave-c0l9 krb5kdc[16446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:34:20.060923 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:38149
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:34:20.395753 16459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:20.396052 16459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:20.396169 16459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:20.403529 16459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:34:20.403654 16459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:20.403714 16459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:20.403795 16459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:34:20.403855 16459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:34:20.413935 16459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:38149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:38149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:20.415271 16459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:20.416566 16459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:20.427965 16467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:20.427999 16465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:20.428040 16464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:20.432081 16459 server_base.cc:1034] running on GCE node
I20260929 11:34:20.432973 16459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:20.434473 16459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:20.435696 16459 hybrid_clock.cc:648] HybridClock initialized: now 1790681660435646 us; error 40 us; skew 500 ppm
I20260929 11:34:20.436061 16459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:34:20 dist-test-slave-c0l9 krb5kdc[16446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681660, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:34:20.439154 16459 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:34:20.440742 16459 webserver.cc:460] Webserver started at http://127.6.9.254:41877/ using document root <none> and password file <none>
I20260929 11:34:20.441447 16459 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:20.441553 16459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:20.441860 16459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:20.444793 16459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "2ef6cc06f5324ac4abd5b0582f3c3626"
format_stamp: "Formatted at 2026-09-29 11:34:20 on dist-test-slave-c0l9"
server_key: "2f0958c1eed9293d38a5c19c7946a650"
server_key_iv: "2ed7d6e35c808e045227d432442c6820"
server_key_version: "encryptionkey@0"
I20260929 11:34:20.445513 16459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "2ef6cc06f5324ac4abd5b0582f3c3626"
format_stamp: "Formatted at 2026-09-29 11:34:20 on dist-test-slave-c0l9"
server_key: "2f0958c1eed9293d38a5c19c7946a650"
server_key_iv: "2ed7d6e35c808e045227d432442c6820"
server_key_version: "encryptionkey@0"
I20260929 11:34:20.450754 16459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:34:20.454320 16474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:20.455711 16459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:34:20.455874 16459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "2ef6cc06f5324ac4abd5b0582f3c3626"
format_stamp: "Formatted at 2026-09-29 11:34:20 on dist-test-slave-c0l9"
server_key: "2f0958c1eed9293d38a5c19c7946a650"
server_key_iv: "2ed7d6e35c808e045227d432442c6820"
server_key_version: "encryptionkey@0"
I20260929 11:34:20.456059 16459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:20.485983 16459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:20.490211 16459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:20.490496 16459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:20.508098 16459 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:38149
I20260929 11:34:20.508149 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:38149 every 8 connection(s)
I20260929 11:34:20.509500 16459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:34:20.513145 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16459
I20260929 11:34:20.513383 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:34:20.513410 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:20.514343 6183 external_mini_cluster.cc:1428] Setting key 052372ebc4f30317128febb6536c8c7a
I20260929 11:34:20.530921 16526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: Bootstrap starting.
Sep 29 11:34:20 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:34:20.534981 16526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:20.536403 16526 log.cc:824] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:20.539948 16526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: No bootstrap required, opened a new log
I20260929 11:34:20.543195 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:20.519192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.6.9.254:38149)
0929 11:34:20.519974 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:20.519986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:20.520031 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:34:20.522145 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:20.522181 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:20.522785 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:20.523470 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:20.526038 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:20.527390 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:20.529250 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:20.529981 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:20.530214 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:20.535065 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:20.535075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:20.535104 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:20.535171 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:20.538618 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:20.539572 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:20.539579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:20.539586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:20.539781 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:20.540232 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:20.540236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:20.540238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:20.540907 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:20.541159 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:20.541605 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 11:34:20.542047 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":364,"threads_started":1}
I20260929 11:34:20.548156 16526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } }
I20260929 11:34:20.548540 16526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:20.548642 16526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef6cc06f5324ac4abd5b0582f3c3626, State: Initialized, Role: FOLLOWER
I20260929 11:34:20.549259 16526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } }
I20260929 11:34:20.549490 16526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:34:20.549628 16526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:34:20.549784 16526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:20.552145 16526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } }
I20260929 11:34:20.552801 16526 leader_election.cc:302] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ef6cc06f5324ac4abd5b0582f3c3626; no voters:
I20260929 11:34:20.553429 16526 leader_election.cc:288] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:34:20.553542 16531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:34:20.554199 16531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 2ef6cc06f5324ac4abd5b0582f3c3626, State: Running, Role: LEADER
I20260929 11:34:20.554713 16531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } }
I20260929 11:34:20.555424 16526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:34:20.557569 16533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ef6cc06f5324ac4abd5b0582f3c3626. Latest consensus state: current_term: 1 leader_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } } }
I20260929 11:34:20.558323 16533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:20.557967 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef6cc06f5324ac4abd5b0582f3c3626" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38149 } } }
I20260929 11:34:20.559476 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626 [sys.catalog]: This master's current role is: LEADER
I20260929 11:34:20.565240 16541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:34:20.570557 16541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:34:20.579246 16541 catalog_manager.cc:1347] Generated new cluster ID: 070e430375a8471683cbd31ed707464d
I20260929 11:34:20.579377 16541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:34:20.597196 16541 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:34:20.598538 16541 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:34:20.606498 16541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: Generated new TSK 0
I20260929 11:34:20.607335 16541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:34:20.674281 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:34:20.999938 16552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:21.000236 16552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:21.000353 16552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:21.007830 16552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:21.007948 16552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:21.008090 16552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:34:21.018095 16552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:21.019423 16552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:21.020784 16552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:21.033908 16557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:21.033957 16558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:21.033979 16560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:21.034782 16552 server_base.cc:1034] running on GCE node
I20260929 11:34:21.035391 16552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:21.036391 16552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:21.037624 16552 hybrid_clock.cc:648] HybridClock initialized: now 1790681661037568 us; error 43 us; skew 500 ppm
I20260929 11:34:21.037974 16552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:34:21 dist-test-slave-c0l9 krb5kdc[16446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:34:21.041267 16552 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:34:21.042793 16552 webserver.cc:460] Webserver started at http://127.6.9.193:38149/ using document root <none> and password file <none>
I20260929 11:34:21.043471 16552 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:21.043591 16552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:21.043883 16552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:21.046918 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "00c3a0e08c6b41c99d47a19f868acb6f"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "2f7ca3a9c68ad10a77cda29484c87808"
server_key_iv: "f19b7a1ee4981db74ad1874a8a8238a0"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.047670 16552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "00c3a0e08c6b41c99d47a19f868acb6f"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "2f7ca3a9c68ad10a77cda29484c87808"
server_key_iv: "f19b7a1ee4981db74ad1874a8a8238a0"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.052923 16552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:34:21.056499 16567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.057964 16552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:21.058135 16552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "00c3a0e08c6b41c99d47a19f868acb6f"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "2f7ca3a9c68ad10a77cda29484c87808"
server_key_iv: "f19b7a1ee4981db74ad1874a8a8238a0"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.058331 16552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:21.086030 16552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:21.090174 16552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:21.090463 16552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:21.091416 16552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:21.092808 16552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:21.092886 16552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.093003 16552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:21.093053 16552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.123142 16552 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:41619
I20260929 11:34:21.123195 16679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:41619 every 8 connection(s)
I20260929 11:34:21.124385 16552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:34:21.129837 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16552
I20260929 11:34:21.130048 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:34:21.130810 6183 external_mini_cluster.cc:1428] Setting key 05568983eca0fb205de788beaee25222
Sep 29 11:34:21 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:34:21.146556 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:21.126898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:42079 (local address 127.6.9.254:38149)
0929 11:34:21.127332 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:21.127336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:21.128419 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 11:34:21.130523 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:21.130528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:21.130598 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:21.130697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:21.132386 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.133244 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.135529 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.135953 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.136062 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:21.140371 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:21.140376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:21.140380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:21.140427 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:21.143069 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:21.144053 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:21.144058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:21.144060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:21.144193 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:21.144760 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:21.144765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:21.144767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:21.145071 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:21.145210 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:21.146102 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 11:34:21.146302 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":205,"threads_started":1}
I20260929 11:34:21.147878 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:21.127435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38149 (local address 127.6.9.193:42079)
0929 11:34:21.128228 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:21.128266 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:21.130202 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:21.130970 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:21.130982 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:21.131660 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:21.132209 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:21.132228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.133413 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.133418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:21.135287 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:21.135310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.136177 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.136183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:21.136462 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:21.137198 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:21.137224 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:21.140051 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:21.143257 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:21.143266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:21.143287 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:21.143910 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:21.144295 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:21.144298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:21.144302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:21.144561 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:21.145196 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:21.145203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:21.145635 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 11:34:21.145967 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 11:34:21.146513 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":307,"threads_started":1}
I20260929 11:34:21.149693 16680 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38149
I20260929 11:34:21.150017 16680 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:21.150867 16680 heartbeater.cc:511] Master 127.6.9.254:38149 requested a full tablet report, sending...
I20260929 11:34:21.153520 16491 ts_manager.cc:194] Registered new tserver with Master: 00c3a0e08c6b41c99d47a19f868acb6f (127.6.9.193:41619)
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:34:21.155494 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:42079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:34:21.183739 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:34:21.504957 16686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:21.505252 16686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:21.505389 16686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:21.512851 16686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:21.512980 16686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:21.513132 16686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
W20260929 11:34:21.515983 16522 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.6.9.254:38149
I20260929 11:34:21.522912 16686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:21.524140 16686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:21.525470 16686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:21.537919 16694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:21.538085 16691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:21.537946 16692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:21.538544 16686 server_base.cc:1034] running on GCE node
I20260929 11:34:21.539207 16686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:21.540222 16686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:21.541419 16686 hybrid_clock.cc:648] HybridClock initialized: now 1790681661541368 us; error 48 us; skew 500 ppm
I20260929 11:34:21.541738 16686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:34:21 dist-test-slave-c0l9 krb5kdc[16446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:34:21.544868 16686 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:34:21.546303 16686 webserver.cc:460] Webserver started at http://127.6.9.194:44895/ using document root <none> and password file <none>
I20260929 11:34:21.546957 16686 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:21.547056 16686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:21.547298 16686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:21.550182 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "0e5584dc02264bc4a04ee32763e62978"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "a80d9522a5ec3735bb49a4cd1d970ae2"
server_key_iv: "f95c563445e9e590656cfe8178339a09"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.550918 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "0e5584dc02264bc4a04ee32763e62978"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "a80d9522a5ec3735bb49a4cd1d970ae2"
server_key_iv: "f95c563445e9e590656cfe8178339a09"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.556221 16686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 11:34:21.559825 16701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.561431 16686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:34:21.561614 16686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "0e5584dc02264bc4a04ee32763e62978"
format_stamp: "Formatted at 2026-09-29 11:34:21 on dist-test-slave-c0l9"
server_key: "a80d9522a5ec3735bb49a4cd1d970ae2"
server_key_iv: "f95c563445e9e590656cfe8178339a09"
server_key_version: "encryptionkey@0"
I20260929 11:34:21.561820 16686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:21.574961 16686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:21.579244 16686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:21.579492 16686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:21.580417 16686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:21.581785 16686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:21.581862 16686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.581950 16686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:21.581995 16686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:21.611596 16686 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42479
I20260929 11:34:21.611641 16813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42479 every 8 connection(s)
I20260929 11:34:21.612851 16686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:34:21.614760 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16686
I20260929 11:34:21.614995 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:34:21.615415 6183 external_mini_cluster.cc:1428] Setting key 8227bf088fc61d1f91638ee737bd20c8
Sep 29 11:34:21 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681661, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:34:21.634805 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:21.615598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:55721 (local address 127.6.9.254:38149)
0929 11:34:21.615804 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:21.615810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:21.617081 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 11:34:21.619486 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:21.619492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:21.619566 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:21.619654 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:21.621663 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.622383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.624488 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.625110 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.625207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:21.628787 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:21.628794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:21.628798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:21.628836 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:21.631680 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:21.632408 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:21.632412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:21.632415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:21.632559 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:21.633051 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:21.633056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:21.633059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:21.633386 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:21.633522 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:21.634205 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 11:34:21.634437 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:34:21.635668 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:21.616161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38149 (local address 127.6.9.194:55721)
0929 11:34:21.616936 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:21.616980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:21.619145 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:21.619918 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:21.619930 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:21.620639 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:21.621451 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:21.621474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.622535 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.622539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:21.624343 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:21.624357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:21.625327 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:21.625334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:21.625596 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:21.626226 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:21.626250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:21.628515 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:21.631847 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:21.631855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:21.631870 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:21.632294 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:21.632727 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:21.632732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:21.632735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:21.632941 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:21.633497 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:21.633503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:21.633842 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 11:34:21.634077 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 11:34:21.634501 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":299,"threads_started":1}
I20260929 11:34:21.637224 16814 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38149
I20260929 11:34:21.637542 16814 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:21.638278 16814 heartbeater.cc:511] Master 127.6.9.254:38149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:34:21.639711 16491 ts_manager.cc:194] Registered new tserver with Master: 0e5584dc02264bc4a04ee32763e62978 (127.6.9.194:42479)
I20260929 11:34:21.640533 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:55721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:34:21.666303 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:34:21.991762 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:34:21.992079 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:34:21.992226 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:34:21.999656 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:34:21.999814 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:34:21.999969 16819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:34:22.009999 16819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:34:22.011350 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:34:22.012807 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:34:22.026507 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:22.026427 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:34:22.026456 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:34:22.027763 16819 server_base.cc:1034] running on GCE node
I20260929 11:34:22.028421 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:34:22.029488 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:34:22.030714 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790681662030646 us; error 68 us; skew 500 ppm
I20260929 11:34:22.031059 16819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681662, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:34:22.034340 16819 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:34:22.036085 16819 webserver.cc:460] Webserver started at http://127.6.9.195:38349/ using document root <none> and password file <none>
I20260929 11:34:22.036849 16819 fs_manager.cc:359] Metadata directory not provided
I20260929 11:34:22.036978 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:34:22.037278 16819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:34:22.040527 16819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "884a28dc693a4657adc8043dc5eced45"
format_stamp: "Formatted at 2026-09-29 11:34:22 on dist-test-slave-c0l9"
server_key: "fa6d994c2de4d3fea2dfb8f6c7677918"
server_key_iv: "eaa709df4b8a4ededb9711fb1acfb5e0"
server_key_version: "encryptionkey@0"
I20260929 11:34:22.041560 16819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "884a28dc693a4657adc8043dc5eced45"
format_stamp: "Formatted at 2026-09-29 11:34:22 on dist-test-slave-c0l9"
server_key: "fa6d994c2de4d3fea2dfb8f6c7677918"
server_key_iv: "eaa709df4b8a4ededb9711fb1acfb5e0"
server_key_version: "encryptionkey@0"
I20260929 11:34:22.047173 16819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:34:22.051031 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:22.052516 16819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:34:22.052742 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "884a28dc693a4657adc8043dc5eced45"
format_stamp: "Formatted at 2026-09-29 11:34:22 on dist-test-slave-c0l9"
server_key: "fa6d994c2de4d3fea2dfb8f6c7677918"
server_key_iv: "eaa709df4b8a4ededb9711fb1acfb5e0"
server_key_version: "encryptionkey@0"
I20260929 11:34:22.052964 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:34:22.083081 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:34:22.087396 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:34:22.087709 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:34:22.088644 16819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:34:22.089998 16819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:34:22.090070 16819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:22.090159 16819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:34:22.090199 16819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:34:22.122289 16819 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:39109
I20260929 11:34:22.122398 16946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:39109 every 8 connection(s)
I20260929 11:34:22.124085 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:34:22.128561 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 16819
I20260929 11:34:22.128786 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:34:22.129308 6183 external_mini_cluster.cc:1428] Setting key d047b36607cef9d488f592dced4d5332
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681662, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:34:22.145632 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.126920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:39113 (local address 127.6.9.254:38149)
0929 11:34:22.127139 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.127146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.128495 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 11:34:22.130626 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.130630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.130759 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.130871 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:22.132919 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.133660 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.135329 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.135784 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.135888 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.139488 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:22.139494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:22.139498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:22.139537 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:22.142134 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.143014 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.143019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.143021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.143166 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.143660 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.143666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.143670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.144080 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:22.144206 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.145154 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.145380 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:34:22.146590 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.127369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38149 (local address 127.6.9.195:39113)
0929 11:34:22.128303 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.128368 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.130290 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.131193 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.131206 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.131959 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:34:22.132724 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.132756 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.133839 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.133853 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.135172 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.135185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.136080 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.136087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.136306 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.137026 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:34:22.137055 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:34:22.139217 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:34:22.142306 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:22.142314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:22.142333 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:22.142814 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:22.143290 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:34:22.143294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:34:22.143297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:34:22.143538 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:34:22.144227 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:34:22.144235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:34:22.144725 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.144992 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.145520 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":429,"threads_started":1}
I20260929 11:34:22.147892 16947 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38149
I20260929 11:34:22.148197 16947 heartbeater.cc:464] Registering TS with master...
I20260929 11:34:22.148998 16947 heartbeater.cc:511] Master 127.6.9.254:38149 requested a full tablet report, sending...
I20260929 11:34:22.150508 16491 ts_manager.cc:194] Registered new tserver with Master: 884a28dc693a4657adc8043dc5eced45 (127.6.9.195:39109)
I20260929 11:34:22.151321 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:39113
I20260929 11:34:22.157927 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:34:22.159042 16680 heartbeater.cc:503] Master 127.6.9.254:38149 was elected leader, sending a full tablet report...
I20260929 11:34:22.175519 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.162844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.6.9.254:38149)
0929 11:34:22.163076 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.163083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.163522 (+ 439us) server_negotiation.cc:407] Connection header received
0929 11:34:22.163669 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.163672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.163739 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.163818 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:22.165362 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.166334 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.167684 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.168105 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.168202 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.170380 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:22.170385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:22.170388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:22.170423 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:22.172915 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.173652 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.173656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.173658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.173829 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.174200 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.174202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.174204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.174526 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:22.174713 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.175090 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.175275 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:34:22.180135 16491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:34:22.187386 16491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:34:22.221181 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.199950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41619 (local address 127.0.0.1:41656)
0929 11:34:22.210477 (+ 10527us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.210547 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.210792 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.212687 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.212695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.212851 (+ 156us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:34:22.213264 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.213277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.215321 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.215327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.217991 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.218015 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.220007 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.220014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.220263 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.220322 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.220471 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.220717 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10173,"mutex_wait_us":6,"thread_start_us":11770,"threads_started":1}
I20260929 11:34:22.221304 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.200514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42479 (local address 127.0.0.1:35086)
0929 11:34:22.212038 (+ 11524us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.212057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.212181 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.213101 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.213106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.213132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:34:22.213604 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.213616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.215403 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.215407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.218367 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.218378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.220344 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.220354 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.220546 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.220583 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.220709 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.220841 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11402,"mutex_wait_us":716,"thread_start_us":9439,"threads_started":1}
I20260929 11:34:22.221904 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.200951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.6.9.194:42479)
0929 11:34:22.211717 (+ 10766us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.211727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.212318 (+ 591us) server_negotiation.cc:407] Connection header received
0929 11:34:22.212419 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.212426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.212652 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.212869 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:34:22.213781 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.215241 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.218526 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.219908 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.220020 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.220173 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.221211 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.221564 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10591,"thread_start_us":10455,"threads_started":1}
I20260929 11:34:22.221977 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.200133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.6.9.193:41619)
0929 11:34:22.211327 (+ 11194us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.211338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.211374 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:34:22.211476 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.211484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.212188 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.212386 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:34:22.213495 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.215128 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.218239 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.219775 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.219932 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.220190 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.221465 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.221637 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10991,"thread_start_us":10820,"threads_started":1}
I20260929 11:34:22.225422 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.212853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:39109 (local address 127.0.0.1:53810)
0929 11:34:22.216383 (+ 3530us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.216402 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.216525 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.217660 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.217664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.217688 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:34:22.218142 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.218154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.219635 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.219639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.221935 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.221950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.224639 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.224646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.224847 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.224879 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.224975 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.225111 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3366,"thread_start_us":3212,"threads_started":1}
I20260929 11:34:22.225495 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.212580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.6.9.195:39109)
0929 11:34:22.216752 (+ 4172us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.216761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.216797 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:34:22.216894 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.216902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.217178 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.217396 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:34:22.218373 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.219489 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.223004 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.224435 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.224571 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.224833 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.225023 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.225178 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3983,"thread_start_us":3839,"threads_started":1}
I20260929 11:34:22.225749 16749 tablet_service.cc:1467] Processing CreateTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 (DEFAULT_TABLE table=test-table [id=b3544b41d0c6482c886ada3807bb90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:34:22.225746 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 (DEFAULT_TABLE table=test-table [id=b3544b41d0c6482c886ada3807bb90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:34:22.227056 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c60cc28a02477ab7324c985a36c3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:22.227092 16749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c60cc28a02477ab7324c985a36c3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:22.229489 16882 tablet_service.cc:1467] Processing CreateTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 (DEFAULT_TABLE table=test-table [id=b3544b41d0c6482c886ada3807bb90c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:34:22.231236 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77c60cc28a02477ab7324c985a36c3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:34:22.238351 16965 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Bootstrap starting.
I20260929 11:34:22.241354 16966 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Bootstrap starting.
I20260929 11:34:22.242918 16965 tablet_bootstrap.cc:654] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:22.244243 16965 log.cc:824] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:22.245325 16966 tablet_bootstrap.cc:654] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:22.245353 16967 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Bootstrap starting.
I20260929 11:34:22.247056 16966 log.cc:824] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:22.248569 16967 tablet_bootstrap.cc:654] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Neither blocks nor log segments found. Creating new log.
I20260929 11:34:22.249711 16965 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: No bootstrap required, opened a new log
I20260929 11:34:22.249918 16967 log.cc:824] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Log is configured to *not* fsync() on all Append() calls
I20260929 11:34:22.250026 16965 ts_tablet_manager.cc:1397] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260929 11:34:22.250093 16966 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: No bootstrap required, opened a new log
I20260929 11:34:22.250356 16966 ts_tablet_manager.cc:1397] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 11:34:22.255764 16967 tablet_bootstrap.cc:492] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: No bootstrap required, opened a new log
I20260929 11:34:22.256102 16967 ts_tablet_manager.cc:1397] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 11:34:22.260104 16965 raft_consensus.cc:348] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.260694 16965 raft_consensus.cc:374] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:22.260814 16965 raft_consensus.cc:729] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c3a0e08c6b41c99d47a19f868acb6f, State: Initialized, Role: FOLLOWER
I20260929 11:34:22.261335 16966 raft_consensus.cc:348] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.261520 16965 consensus_queue.cc:260] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.261818 16966 raft_consensus.cc:374] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:22.261932 16966 raft_consensus.cc:729] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5584dc02264bc4a04ee32763e62978, State: Initialized, Role: FOLLOWER
I20260929 11:34:22.262567 16966 consensus_queue.cc:260] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.263881 16814 heartbeater.cc:503] Master 127.6.9.254:38149 was elected leader, sending a full tablet report...
I20260929 11:34:22.264562 16966 ts_tablet_manager.cc:1428] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 11:34:22.265185 16965 ts_tablet_manager.cc:1428] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 11:34:22.265623 16971 raft_consensus.cc:482] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:34:22.265911 16971 raft_consensus.cc:504] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.268217 16971 leader_election.cc:288] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 884a28dc693a4657adc8043dc5eced45 (127.6.9.195:39109), 0e5584dc02264bc4a04ee32763e62978 (127.6.9.194:42479)
I20260929 11:34:22.270391 16967 raft_consensus.cc:348] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.270946 16967 raft_consensus.cc:374] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:34:22.271028 16967 raft_consensus.cc:729] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884a28dc693a4657adc8043dc5eced45, State: Initialized, Role: FOLLOWER
I20260929 11:34:22.271797 16967 consensus_queue.cc:260] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.277854 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.269050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58269 (local address 127.6.9.195:39109)
0929 11:34:22.269274 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.269281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.269775 (+ 494us) server_negotiation.cc:407] Connection header received
0929 11:34:22.270116 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.270121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.270222 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.270289 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:34:22.271218 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.272317 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.275263 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.276622 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.276760 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.276870 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.277426 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.277585 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:34:22.277971 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.268489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:39109 (local address 127.6.9.193:58269)
0929 11:34:22.269668 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.269695 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.269900 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.270507 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.270511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.270538 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:34:22.271055 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.271068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.272531 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.272541 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.275004 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.275023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.277061 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.277067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.277250 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.277282 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.277382 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.277521 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":853,"threads_started":1}
I20260929 11:34:22.279076 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c60cc28a02477ab7324c985a36c3d0" candidate_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884a28dc693a4657adc8043dc5eced45" is_pre_election: true
I20260929 11:34:22.279711 16902 raft_consensus.cc:2454] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c3a0e08c6b41c99d47a19f868acb6f in term 0.
I20260929 11:34:22.280544 16569 leader_election.cc:302] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c3a0e08c6b41c99d47a19f868acb6f, 884a28dc693a4657adc8043dc5eced45; no voters:
I20260929 11:34:22.281040 16971 raft_consensus.cc:2781] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:34:22.281224 16971 raft_consensus.cc:482] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:34:22.281330 16971 raft_consensus.cc:3037] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:22.284238 16971 raft_consensus.cc:504] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.285501 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.270890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:45937 (local address 127.6.9.194:42479)
0929 11:34:22.277276 (+ 6386us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.277283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.278426 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 11:34:22.278650 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.278656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.278724 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.278776 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:34:22.279252 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.280319 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.282542 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.283882 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.284035 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.284143 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.285048 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.285217 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6214}
I20260929 11:34:22.285530 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c60cc28a02477ab7324c985a36c3d0" candidate_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884a28dc693a4657adc8043dc5eced45"
I20260929 11:34:22.285760 16902 raft_consensus.cc:3037] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:22.286303 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.270408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42479 (local address 127.6.9.193:45937)
0929 11:34:22.278347 (+ 7939us) negotiation.cc:107] Waiting for socket to connect
0929 11:34:22.278364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:34:22.278525 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:34:22.278936 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:34:22.278941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:34:22.278964 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:34:22.279117 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.279126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.280537 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.280543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.282340 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:34:22.282355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.284282 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.284289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:34:22.284786 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.284826 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:34:22.285833 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0929 11:34:22.286045 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7834,"threads_started":1}
I20260929 11:34:22.289099 16971 leader_election.cc:288] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [CANDIDATE]: Term 1 election: Requested vote from peers 884a28dc693a4657adc8043dc5eced45 (127.6.9.195:39109), 0e5584dc02264bc4a04ee32763e62978 (127.6.9.194:42479)
I20260929 11:34:22.289395 16902 raft_consensus.cc:2454] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c3a0e08c6b41c99d47a19f868acb6f in term 1.
I20260929 11:34:22.290050 16569 leader_election.cc:302] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c3a0e08c6b41c99d47a19f868acb6f, 884a28dc693a4657adc8043dc5eced45; no voters:
I20260929 11:34:22.290421 16971 raft_consensus.cc:2781] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:34:22.290693 16947 heartbeater.cc:503] Master 127.6.9.254:38149 was elected leader, sending a full tablet report...
I20260929 11:34:22.291115 16768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c60cc28a02477ab7324c985a36c3d0" candidate_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5584dc02264bc4a04ee32763e62978"
I20260929 11:34:22.291169 16769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77c60cc28a02477ab7324c985a36c3d0" candidate_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5584dc02264bc4a04ee32763e62978" is_pre_election: true
I20260929 11:34:22.291602 16768 raft_consensus.cc:3037] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:34:22.293501 16967 ts_tablet_manager.cc:1428] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Time spent starting tablet: real 0.037s user 0.013s sys 0.013s
I20260929 11:34:22.295039 16768 raft_consensus.cc:2454] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c3a0e08c6b41c99d47a19f868acb6f in term 1.
I20260929 11:34:22.295806 16971 raft_consensus.cc:686] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 1 LEADER]: Becoming Leader. State: Replica: 00c3a0e08c6b41c99d47a19f868acb6f, State: Running, Role: LEADER
I20260929 11:34:22.296461 16971 consensus_queue.cc:237] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } }
I20260929 11:34:22.301324 16491 catalog_manager.cc:5495] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c3a0e08c6b41c99d47a19f868acb6f (127.6.9.193). New cstate: current_term: 1 leader_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c3a0e08c6b41c99d47a19f868acb6f" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260929 11:34:22.340477 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.335462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.6.9.193:41619)
0929 11:34:22.335684 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.335689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.335707 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:34:22.335882 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.335886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.335965 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.336091 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:34:22.336396 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.337160 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.338434 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.338838 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.338928 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.339392 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:34:22.339913 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:34:22.340024 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.340149 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.340220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:34:22.348738 16768 raft_consensus.cc:1261] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 1 FOLLOWER]: Refusing update from remote peer 00c3a0e08c6b41c99d47a19f868acb6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:34:22.348837 16902 raft_consensus.cc:1261] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 1 FOLLOWER]: Refusing update from remote peer 00c3a0e08c6b41c99d47a19f868acb6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:34:22.349684 16976 consensus_queue.cc:1035] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5584dc02264bc4a04ee32763e62978" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:34:22.350023 16971 consensus_queue.cc:1035] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "884a28dc693a4657adc8043dc5eced45" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:34:22.363799 16980 mvcc.cc:205] Tried to move back new op lower bound from 7334632088971005952 to 7334632088770224128. Current Snapshot: MvccSnapshot[applied={T|T < 7334632088971005952}]
I20260929 11:34:22.376300 16490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:34:22.376647 16490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:34:22.380159 16490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ef6cc06f5324ac4abd5b0582f3c3626: Sending DeleteTablet for 3 replicas of tablet 77c60cc28a02477ab7324c985a36c3d0
I20260929 11:34:22.381317 16749 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:34:22 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:35086
I20260929 11:34:22.381414 16882 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:34:22 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:53810
I20260929 11:34:22.382086 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet 77c60cc28a02477ab7324c985a36c3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:34:22 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:41656
I20260929 11:34:22.382174 16989 tablet_replica.cc:330] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: stopping tablet replica
I20260929 11:34:22.382330 16990 tablet_replica.cc:330] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: stopping tablet replica
I20260929 11:34:22.382717 16989 raft_consensus.cc:2229] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:34:22.383100 16990 raft_consensus.cc:2229] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:34:22.383353 16989 raft_consensus.cc:2258] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:34:22.383559 16990 raft_consensus.cc:2258] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:34:22.385166 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16552
I20260929 11:34:22.386003 16989 ts_tablet_manager.cc:1905] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:34:22.386021 16990 ts_tablet_manager.cc:1905] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:34:22.392822 16990 ts_tablet_manager.cc:1918] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:34:22.392985 16990 log.cc:1196] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/77c60cc28a02477ab7324c985a36c3d0
I20260929 11:34:22.393121 16989 ts_tablet_manager.cc:1918] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:34:22.393376 16989 log.cc:1196] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/77c60cc28a02477ab7324c985a36c3d0
I20260929 11:34:22.393496 16990 ts_tablet_manager.cc:1939] T 77c60cc28a02477ab7324c985a36c3d0 P 884a28dc693a4657adc8043dc5eced45: Deleting consensus metadata
I20260929 11:34:22.393983 16989 ts_tablet_manager.cc:1939] T 77c60cc28a02477ab7324c985a36c3d0 P 0e5584dc02264bc4a04ee32763e62978: Deleting consensus metadata
I20260929 11:34:22.395373 16476 catalog_manager.cc:4841] TS 884a28dc693a4657adc8043dc5eced45 (127.6.9.195:39109): tablet 77c60cc28a02477ab7324c985a36c3d0 (table test-table [id=b3544b41d0c6482c886ada3807bb90c5]) successfully deleted
I20260929 11:34:22.395709 16478 catalog_manager.cc:4841] TS 0e5584dc02264bc4a04ee32763e62978 (127.6.9.194:42479): tablet 77c60cc28a02477ab7324c985a36c3d0 (table test-table [id=b3544b41d0c6482c886ada3807bb90c5]) successfully deleted
I20260929 11:34:22.395974 16991 tablet_replica.cc:330] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: stopping tablet replica
I20260929 11:34:22.396430 16991 raft_consensus.cc:2229] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:34:22.397125 16991 raft_consensus.cc:2258] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:34:22.399291 16991 ts_tablet_manager.cc:1905] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:34:22.409013 16991 ts_tablet_manager.cc:1918] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:34:22.409220 16991 log.cc:1196] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/77c60cc28a02477ab7324c985a36c3d0
I20260929 11:34:22.409818 16991 ts_tablet_manager.cc:1939] T 77c60cc28a02477ab7324c985a36c3d0 P 00c3a0e08c6b41c99d47a19f868acb6f: Deleting consensus metadata
I20260929 11:34:22.411759 16476 catalog_manager.cc:4841] TS 00c3a0e08c6b41c99d47a19f868acb6f (127.6.9.193:41619): tablet 77c60cc28a02477ab7324c985a36c3d0 (table test-table [id=b3544b41d0c6482c886ada3807bb90c5]) successfully deleted
I20260929 11:34:22.412561 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.387842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.6.9.193:41619)
0929 11:34:22.388019 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.388025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.396726 (+ 8701us) server_negotiation.cc:407] Connection header received
0929 11:34:22.397078 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.397084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.397172 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.397274 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:22.398629 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.399386 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.401223 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.402510 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.402651 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.406428 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:22.406440 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:22.406455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:22.406496 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:22.409655 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.410401 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.410409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.410412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.410574 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.410959 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.410966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.410968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.411291 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:22.411604 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.411969 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.412223 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:34:22.413260 16675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:22.512523 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16552
I20260929 11:34:22.535816 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16686
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:34:22.553267 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.536901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.6.9.194:42479)
0929 11:34:22.537061 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.537066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.537084 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:34:22.537329 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.537336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.537459 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.537744 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:22.539020 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.539971 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.541386 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.541802 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.541920 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.546394 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:22.546411 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:22.546429 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:22.546470 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:22.549806 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.550751 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.550760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.550762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.550910 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.551410 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.551415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.551417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.551878 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:22.552115 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.552776 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.552990 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:34:22.554409 16809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:22.643265 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16686
I20260929 11:34:22.657950 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16819
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:34:22.672413 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:34:22.658657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.6.9.195:39109)
0929 11:34:22.658854 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:34:22.658860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:34:22.658886 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:34:22.659092 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:34:22.659096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:34:22.659180 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:34:22.659255 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:34:22.660564 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.661381 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.662561 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:34:22.662965 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:34:22.663063 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:34:22.667160 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:34:22.667177 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:34:22.667195 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:34:22.667238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:34:22.669874 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.670577 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.670585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.670587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.670734 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:34:22.671060 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:34:22.671063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:34:22.671066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:34:22.671431 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:34:22.671669 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:34:22.671891 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 11:34:22.672084 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:34:22.673127 16942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:22.760759 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16819
I20260929 11:34:22.773672 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 16459
I20260929 11:34:22.774734 16521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:34:22.882653 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 16459
2026-09-29T11:34:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4053 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16999](info): set up 2 sockets
Sep 29 11:34:22 dist-test-slave-c0l9 krb5kdc[16999](info): commencing operation
krb5kdc: starting...
W20260929 11:34:23.893234 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:34:24 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:34:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangeradmin_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangeradmin_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangerlookup_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangerlookup_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
I20260929 11:34:24.211647 6183 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvWjXHB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/postgres -l logfile start
2026-09-29 11:34:26.695 UTC [17029] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:34:26.695 UTC [17029] LOG: listening on IPv4 address "127.6.9.212", port 39161
2026-09-29 11:34:26.699 UTC [17029] LOG: listening on Unix socket "/tmp/.s.PGSQL.39161"
2026-09-29 11:34:26.719 UTC [17032] LOG: database system was shut down at 2026-09-29 11:34:25 UTC
2026-09-29 11:34:26.722 UTC [17029] LOG: database system is ready to accept connections
I20260929 11:34:27.616943 6183 mini_postgres.cc:96] Postgres bound to 39161
2026-09-29 11:34:27.633 UTC [17040] FATAL: database "slave" does not exist
127.6.9.212:39161 - accepting connections
I20260929 11:34:27.634131 6183 mini_ranger.cc:161] Starting Ranger...
I20260929 11:34:27.659212 6183 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:34:28.130663 6183 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:34:28.130790 6183 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin
2026-09-29 11:34:28,427 [I] DB FLAVOR :POSTGRES
2026-09-29 11:34:28,428 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:34:28,428 [I] Checking connection..
2026-09-29 11:34:28,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:34:28,691 [I] Checking connection passed.
2026-09-29 11:34:28,691 [I] --------- Verifying version history table ---------
2026-09-29 11:34:28,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:34:28,939 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:34:28,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:34:29,180 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:34:29,180 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:34:29,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:34:29.428 UTC [17104] WARNING: there is no transaction in progress
2026-09-29 11:34:29,434 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:34:29,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:34:29,694 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:34:29,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:34:29,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:34:29,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:34:30,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c0l9.c.gcp-upstream.internal','N') ;"
2026-09-29 11:34:30,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:34:30,675 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:34:30,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:34:30,937 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:34:30,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:34:31,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:34:31,428 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:34:31,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:34:32.424 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.440 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.458 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.473 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.489 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.505 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.562 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.569 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.584 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.595 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.607 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.622 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.633 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.640 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.647 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:32.654 UTC [17212] WARNING: there is no transaction in progress
2026-09-29 11:34:33,316 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:34:33,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:34:33,568 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:34:33,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:34:33,826 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:34:33,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:34:34,065 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:34:34,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:34:34,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:34:34,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c0l9.c.gcp-upstream.internal';"
2026-09-29 11:34:34,791 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:34:34,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:34:35,049 [I] DB_PATCHES have already been applied
I20260929 11:34:35.054807 6183 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvWjXHB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:34:35.054901 6183 mini_ranger.cc:193] Using host: 127.6.9.212
I20260929 11:34:35.068943 6183 mini_ranger.cc:227] Ranger admin URL: http://127.6.9.212:44395
Sep 29, 2026 11:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:34:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:34:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:34:36.107587 6183 mini_ranger.cc:160] Time spent starting Ranger: real 8.473s user 0.003s sys 0.037s
Sep 29, 2026 11:34:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:34:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:34:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:34:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:34:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:34:58.749683 6183 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangerkms_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/rangerkms_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.9.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.9.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:34:58 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681698, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:34:58.918731 6183 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:34:58.946250 6183 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:34:59.820822 6183 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:34:59.820971 6183 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms
I20260929 11:34:59.821095 6183 mini_ranger_kms.cc:227] Using postgres at 127.6.9.212:39161
2026-09-29 11:35:00,259 [I] DB FLAVOR :POSTGRES
2026-09-29 11:35:00,259 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:35:00,259 [I] Checking connection
2026-09-29 11:35:00,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:35:00,488 [I] connection success
2026-09-29 11:35:00,489 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:35:00,489 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:35:00,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:35:00,751 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:35:00,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:35:00,751 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:35:00,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:39161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:35:01,028 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:35:01.479238 6183 mini_ranger_kms.cc:310] Created kms service
I20260929 11:35:01.669878 6183 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:35:01.717736 6183 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:35:01 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681698, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:02.167837 6183 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:35:02.168136 6183 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:35:02.168184 6183 mini_ranger_kms.cc:240] Using host: 127.6.9.212
I20260929 11:35:02.185169 6183 mini_ranger_kms.cc:276] Ranger KMS PID: 17393
I20260929 11:35:02.185365 6183 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.9.212:51395
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 11:35:04.232409 6183 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.314s user 0.127s sys 0.331s
I20260929 11:35:04.232707 6183 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:35:04.232933 6183 mini_ranger_kms.cc:401] 127.6.9.212:51395/kms/v1/keys
Sep 29 11:35:04 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 11:35:04 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
Sep 29 11:35:05 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
Sep 29 11:35:07 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:35:07.497104 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46099
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46099
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51395/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:35:07.829396 17443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:35:07.829706 17443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:35:07.829815 17443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:35:07.837116 17443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:35:07.837239 17443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:35:07.837282 17443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:35:07.837337 17443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:35:07.837381 17443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:35:07.847460 17443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.9.212:51395/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46099
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:35:07.848749 17443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:35:07.850031 17443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:35:07.861969 17448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:07.861976 17451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:07.861956 17449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:07.862850 17443 server_base.cc:1034] running on GCE node
I20260929 11:35:07.863735 17443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:35:07.865319 17443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:35:07.866601 17443 hybrid_clock.cc:648] HybridClock initialized: now 1790681707866522 us; error 52 us; skew 500 ppm
I20260929 11:35:07.867024 17443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:35:07 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:35:07.873268 17443 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:35:07.874972 17443 webserver.cc:460] Webserver started at http://127.6.9.254:40987/ using document root <none> and password file <none>
I20260929 11:35:07.875699 17443 fs_manager.cc:359] Metadata directory not provided
I20260929 11:35:07.875808 17443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:35:07.876075 17443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:35:07 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:07.939421 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "935d46617c5d409991fe51c9d02b310d"
format_stamp: "Formatted at 2026-09-29 11:35:07 on dist-test-slave-c0l9"
server_key: "f9d656dba12f9d542efad45522f2305d"
server_key_iv: "84d43e28be9fc7cab54654da408de6aa"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:07.940785 17443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "935d46617c5d409991fe51c9d02b310d"
format_stamp: "Formatted at 2026-09-29 11:35:07 on dist-test-slave-c0l9"
server_key: "f9d656dba12f9d542efad45522f2305d"
server_key_iv: "84d43e28be9fc7cab54654da408de6aa"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:07.948921 17443 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 11:35:07.970309 17458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:07.972138 17443 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:35:07.972355 17443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "935d46617c5d409991fe51c9d02b310d"
format_stamp: "Formatted at 2026-09-29 11:35:07 on dist-test-slave-c0l9"
server_key: "f9d656dba12f9d542efad45522f2305d"
server_key_iv: "84d43e28be9fc7cab54654da408de6aa"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:07.972640 17443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:35:07.998001 17443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:35:08.003211 17443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:35:08.003567 17443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:35:08.023623 17443 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46099
I20260929 11:35:08.023644 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46099 every 8 connection(s)
I20260929 11:35:08.025442 17443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:35:08.029716 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:35:08.034196 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 17443
I20260929 11:35:08.034402 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:35:08.035202 6183 external_mini_cluster.cc:1428] Setting key d3fc7cf18b05b77e04d0fe7f08d81a77
I20260929 11:35:08.043637 17510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: Bootstrap starting.
I20260929 11:35:08.048475 17510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: Neither blocks nor log segments found. Creating new log.
I20260929 11:35:08.050414 17510 log.cc:824] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:35:08.053517 17510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: No bootstrap required, opened a new log
I20260929 11:35:08.063588 17510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } }
I20260929 11:35:08.064188 17510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:35:08.064308 17510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935d46617c5d409991fe51c9d02b310d, State: Initialized, Role: FOLLOWER
Sep 29 11:35:08 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:35:08.065040 17510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } }
I20260929 11:35:08.065284 17510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:35:08.065448 17510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:35:08.065623 17510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:35:08.069146 17510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } }
I20260929 11:35:08.069902 17510 leader_election.cc:302] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935d46617c5d409991fe51c9d02b310d; no voters:
I20260929 11:35:08.071517 17510 leader_election.cc:288] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:35:08.071732 17515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:35:08.075814 17515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [term 1 LEADER]: Becoming Leader. State: Replica: 935d46617c5d409991fe51c9d02b310d, State: Running, Role: LEADER
I20260929 11:35:08.076561 17515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } }
I20260929 11:35:08.077371 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:08.037498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.6.9.254:46099)
0929 11:35:08.049241 (+ 11743us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:08.049259 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:08.049916 (+ 657us) server_negotiation.cc:407] Connection header received
0929 11:35:08.052459 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:08.052496 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:08.053097 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:08.053545 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:08.055165 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:08.057285 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:08.059339 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:08.060616 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:08.060891 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:08.065938 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:08.065951 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:08.065976 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:08.066034 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:08.070262 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:08.071275 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:08.071285 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:08.071298 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:08.071539 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:08.072104 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:08.072110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:08.072114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:08.073091 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:08.074769 (+ 1678us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:08.075119 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 11:35:08.075679 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11452,"thread_start_us":11227,"threads_started":1}
I20260929 11:35:08.081211 17510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:35:08.085892 17516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935d46617c5d409991fe51c9d02b310d. Latest consensus state: current_term: 1 leader_uuid: "935d46617c5d409991fe51c9d02b310d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } } }
I20260929 11:35:08.085928 17517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "935d46617c5d409991fe51c9d02b310d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935d46617c5d409991fe51c9d02b310d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46099 } } }
I20260929 11:35:08.086351 17516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [sys.catalog]: This master's current role is: LEADER
I20260929 11:35:08.086351 17517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d [sys.catalog]: This master's current role is: LEADER
I20260929 11:35:08.087313 17522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:35:08.094340 17522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:35:08.106571 17522 catalog_manager.cc:1347] Generated new cluster ID: c62424af4789415fafe87041108c902c
I20260929 11:35:08.106721 17522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:35:08.129788 17510 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:35:08.135522 17522 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:35:08.137773 17522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:35:08.157948 17522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: Generated new TSK 0
I20260929 11:35:08.159371 17522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:35:08.980065 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:35:09 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:35:09.404024 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:35:09.404346 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:35:09.404505 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:35:09.414263 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:35:09.414510 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:35:09.414698 17556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:35:09.426367 17556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.9.212:51395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:35:09.427743 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:35:09.429188 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:35:09.453456 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:09.453763 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:09.458282 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:09.458600 17556 server_base.cc:1034] running on GCE node
I20260929 11:35:09.459301 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:35:09.460477 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:35:09.461797 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790681709461739 us; error 63 us; skew 500 ppm
I20260929 11:35:09.462224 17556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:35:09 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:35:09.465837 17556 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:35:09.471451 17556 webserver.cc:460] Webserver started at http://127.6.9.193:46705/ using document root <none> and password file <none>
I20260929 11:35:09.472587 17556 fs_manager.cc:359] Metadata directory not provided
I20260929 11:35:09.472864 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:35:09.473318 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:35:09 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:09.504024 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "d96b1ebe82d64e9785845f123b50df43"
format_stamp: "Formatted at 2026-09-29 11:35:09 on dist-test-slave-c0l9"
server_key: "788cf8f383ce01af378235c4bd0c185a"
server_key_iv: "ef9a7e2b4594c18b95bde8158d59cc87"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:09.505517 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "d96b1ebe82d64e9785845f123b50df43"
format_stamp: "Formatted at 2026-09-29 11:35:09 on dist-test-slave-c0l9"
server_key: "788cf8f383ce01af378235c4bd0c185a"
server_key_iv: "ef9a7e2b4594c18b95bde8158d59cc87"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:09.517696 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.009s
I20260929 11:35:09.538795 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:09.540359 17556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:35:09.540557 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "d96b1ebe82d64e9785845f123b50df43"
format_stamp: "Formatted at 2026-09-29 11:35:09 on dist-test-slave-c0l9"
server_key: "788cf8f383ce01af378235c4bd0c185a"
server_key_iv: "ef9a7e2b4594c18b95bde8158d59cc87"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:09.540830 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:35:09.560763 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:35:09.567314 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:35:09.567762 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:35:09.573562 17556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:35:09.578444 17556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:35:09.578675 17556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:35:09.578900 17556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:35:09.579021 17556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:09.635691 17556 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:37513
I20260929 11:35:09.637233 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:35:09.635691 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:37513 every 8 connection(s)
I20260929 11:35:09.638088 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 17556
I20260929 11:35:09.638312 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:35:09.639128 6183 external_mini_cluster.cc:1428] Setting key 52a6d2d9a9e42b851da81fee97263270
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Sep 29 11:35:09 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:35:09.713274 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:09.658629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:49477 (local address 127.6.9.254:46099)
0929 11:35:09.683898 (+ 25269us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:09.683907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:09.688267 (+ 4360us) server_negotiation.cc:407] Connection header received
0929 11:35:09.692903 (+ 4636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:09.692909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:09.693024 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:09.693122 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:09.695562 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:09.696716 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:09.699356 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:09.700024 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:09.700146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:09.705752 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:09.705759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:09.705763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:09.705805 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:09.709119 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:09.710200 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:09.710206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:09.710209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:09.710379 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:09.710959 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:09.710964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:09.710967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:09.711308 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:09.711431 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:09.712577 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0929 11:35:09.712889 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25081,"thread_start_us":25282,"threads_started":1}
I20260929 11:35:09.715795 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:09.657828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46099 (local address 127.6.9.193:49477)
0929 11:35:09.684805 (+ 26977us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:09.684889 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:09.687873 (+ 2984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:09.693617 (+ 5744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:09.693632 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:09.694307 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:35:09.695306 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:09.695351 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:09.696966 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:09.696972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:09.699145 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:09.699163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:09.700539 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:09.700547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:09.700895 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:09.701617 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:35:09.701641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:35:09.705451 (+ 3810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:35:09.709339 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:09.709349 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:09.709370 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:09.710049 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:09.710543 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:09.710547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:09.710551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:09.710832 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:09.711542 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:35:09.711551 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:35:09.712005 (+ 454us) client_negotiation.cc:769] Sending connection context
0929 11:35:09.713617 (+ 1612us) client_negotiation.cc:240] Negotiation successful
0929 11:35:09.714248 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26497,"thread_start_us":30936,"threads_started":1}
I20260929 11:35:09.718034 17689 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46099
I20260929 11:35:09.718324 17689 heartbeater.cc:464] Registering TS with master...
I20260929 11:35:09.719285 17689 heartbeater.cc:511] Master 127.6.9.254:46099 requested a full tablet report, sending...
I20260929 11:35:09.722290 17475 ts_manager.cc:194] Registered new tserver with Master: d96b1ebe82d64e9785845f123b50df43 (127.6.9.193:37513)
I20260929 11:35:09.724336 17475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:49477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:35:09.741986 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:35:09 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
Sep 29 11:35:10 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681709, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
W20260929 11:35:10.198071 17695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:35:10.198496 17695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:35:10.198705 17695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:35:10.210469 17695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:35:10.210623 17695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:35:10.210863 17695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:35:10.227134 17695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.9.212:51395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:35:10.229022 17695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:35:10.230784 17695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:35:10.249583 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:10.250877 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:10.254750 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:10.255568 17695 server_base.cc:1034] running on GCE node
I20260929 11:35:10.256421 17695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:35:10.273037 17695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:35:10.274422 17695 hybrid_clock.cc:648] HybridClock initialized: now 1790681710274339 us; error 85 us; skew 500 ppm
I20260929 11:35:10.274933 17695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:35:10 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681710, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:35:10.279642 17695 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:35:10.322887 17695 webserver.cc:460] Webserver started at http://127.6.9.194:36341/ using document root <none> and password file <none>
I20260929 11:35:10.324120 17695 fs_manager.cc:359] Metadata directory not provided
I20260929 11:35:10.324398 17695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:35:10.324914 17695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:35:10 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681710, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:10.344491 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "72160a8b5e8a40d4b02114935f7941b9"
format_stamp: "Formatted at 2026-09-29 11:35:10 on dist-test-slave-c0l9"
server_key: "42dd6153f6b85503e3b6098e2f7347d5"
server_key_iv: "7bb415aeaf225c98621150c91072e5cd"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:10.346089 17695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "72160a8b5e8a40d4b02114935f7941b9"
format_stamp: "Formatted at 2026-09-29 11:35:10 on dist-test-slave-c0l9"
server_key: "42dd6153f6b85503e3b6098e2f7347d5"
server_key_iv: "7bb415aeaf225c98621150c91072e5cd"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:10.371304 17695 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.005s sys 0.017s
I20260929 11:35:10.425415 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:10.427340 17695 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.002s sys 0.000s
I20260929 11:35:10.427546 17695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "72160a8b5e8a40d4b02114935f7941b9"
format_stamp: "Formatted at 2026-09-29 11:35:10 on dist-test-slave-c0l9"
server_key: "42dd6153f6b85503e3b6098e2f7347d5"
server_key_iv: "7bb415aeaf225c98621150c91072e5cd"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:10.427781 17695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:35:10.514900 17695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:35:10.520342 17695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:35:10.520782 17695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:35:10.526005 17695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:35:10.537801 17695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:35:10.538004 17695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:10.538188 17695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:35:10.538285 17695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:10.728567 17689 heartbeater.cc:503] Master 127.6.9.254:46099 was elected leader, sending a full tablet report...
I20260929 11:35:11.004820 17695 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42207
I20260929 11:35:11.004853 17831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42207 every 8 connection(s)
I20260929 11:35:11.006129 17695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:35:11.014132 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 17695
I20260929 11:35:11.014319 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:35:11.014843 6183 external_mini_cluster.cc:1428] Setting key 68f74b79dc927f29c99c23a405596dff
Sep 29 11:35:11 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681710, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:35:11.033617 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.008751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:46001 (local address 127.6.9.254:46099)
0929 11:35:11.009271 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.009278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.010249 (+ 971us) server_negotiation.cc:407] Connection header received
0929 11:35:11.012101 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.012108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.012190 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.012315 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:11.014428 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.015253 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.017439 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.017840 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.017950 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.021469 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:11.021475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:11.021479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:11.021517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:11.028887 (+ 7370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.030060 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.030066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.030069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.030236 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.030815 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.030820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.030823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.031180 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:11.031346 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.032943 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.033184 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":237,"threads_started":1}
I20260929 11:35:11.034310 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.009295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46099 (local address 127.6.9.194:46001)
0929 11:35:11.010057 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.010117 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.011793 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.012529 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.012538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.013041 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:35:11.013637 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.013656 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.015433 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.015439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.017290 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.017304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.018067 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.018074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.018297 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.018945 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:35:11.018968 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:35:11.021201 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:35:11.029143 (+ 7942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:11.029155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:11.029177 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:11.029885 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:11.030398 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:11.030403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:11.030407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:11.030678 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:11.031427 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:35:11.031436 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:35:11.031875 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.032170 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.032767 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":307,"threads_started":1}
I20260929 11:35:11.036195 17832 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46099
I20260929 11:35:11.036541 17832 heartbeater.cc:464] Registering TS with master...
I20260929 11:35:11.037500 17832 heartbeater.cc:511] Master 127.6.9.254:46099 requested a full tablet report, sending...
I20260929 11:35:11.039317 17475 ts_manager.cc:194] Registered new tserver with Master: 72160a8b5e8a40d4b02114935f7941b9 (127.6.9.194:42207)
I20260929 11:35:11.040417 17475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:46001
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:35:11.072613 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:35:11.405709 17838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:35:11.406030 17838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:35:11.406178 17838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:35:11.413594 17838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:35:11.413718 17838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:35:11.413877 17838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:35:11.423789 17838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.9.212:51395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:35:11.425107 17838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:35:11.426350 17838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:35:11.440595 17843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:11.441004 17844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:11.441038 17846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:11.441563 17838 server_base.cc:1034] running on GCE node
I20260929 11:35:11.442201 17838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:35:11.443122 17838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:35:11.444346 17838 hybrid_clock.cc:648] HybridClock initialized: now 1790681711444282 us; error 55 us; skew 500 ppm
I20260929 11:35:11.444766 17838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:35:11 dist-test-slave-c0l9 krb5kdc[16999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681711, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:35:11.447889 17838 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:35:11.449477 17838 webserver.cc:460] Webserver started at http://127.6.9.195:41477/ using document root <none> and password file <none>
I20260929 11:35:11.450162 17838 fs_manager.cc:359] Metadata directory not provided
I20260929 11:35:11.450291 17838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:35:11.450624 17838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:35:11 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681711, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:11.465624 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "61f5b07d09cc456b96cf092f16c9aa6b"
format_stamp: "Formatted at 2026-09-29 11:35:11 on dist-test-slave-c0l9"
server_key: "59ffb965247c2e314e9ec571c44331f3"
server_key_iv: "b809f493cfdcd6e48021a7aa8696b5f7"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:11.466455 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "61f5b07d09cc456b96cf092f16c9aa6b"
format_stamp: "Formatted at 2026-09-29 11:35:11 on dist-test-slave-c0l9"
server_key: "59ffb965247c2e314e9ec571c44331f3"
server_key_iv: "b809f493cfdcd6e48021a7aa8696b5f7"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:11.471789 17838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:35:11.487514 17853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:11.489244 17838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:35:11.489450 17838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "61f5b07d09cc456b96cf092f16c9aa6b"
format_stamp: "Formatted at 2026-09-29 11:35:11 on dist-test-slave-c0l9"
server_key: "59ffb965247c2e314e9ec571c44331f3"
server_key_iv: "b809f493cfdcd6e48021a7aa8696b5f7"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:11.489682 17838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:35:11.507611 17838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:35:11.511471 17838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:35:11.511770 17838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:35:11.512769 17838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:35:11.514128 17838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:35:11.514216 17838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:11.514339 17838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:35:11.514392 17838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:11.545342 17838 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:41917
I20260929 11:35:11.545380 17965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:41917 every 8 connection(s)
I20260929 11:35:11.546641 17838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:35:11.547583 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 17838
I20260929 11:35:11.547756 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:35:11.548252 6183 external_mini_cluster.cc:1428] Setting key 73d5934f0e56041b64b4ef5bee691bd9
Sep 29 11:35:11 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681711, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:35:11.566239 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.549328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:36657 (local address 127.6.9.254:46099)
0929 11:35:11.550347 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.550354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.551047 (+ 693us) server_negotiation.cc:407] Connection header received
0929 11:35:11.552853 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.552857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.552923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.553013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:11.554464 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.555224 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.556718 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.557155 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.557248 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.560445 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:11.560451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:11.560456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:11.560498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:11.563455 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.564228 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.564232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.564234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.564356 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.564784 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.564787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.564789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.565057 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:11.565169 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.565819 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.565983 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":745,"threads_started":1}
I20260929 11:35:11.567183 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.550044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46099 (local address 127.6.9.195:36657)
0929 11:35:11.550904 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.550950 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.552554 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.553305 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.553314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.553752 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:35:11.554330 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.554348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.555341 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.555344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.556572 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.556583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.557336 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.557341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.557538 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.558206 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:35:11.558231 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:35:11.560208 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:35:11.563592 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:11.563601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:11.563620 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:11.564133 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:11.564435 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:35:11.564439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:35:11.564441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:35:11.564639 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:35:11.565175 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:35:11.565183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:35:11.565495 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.565754 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.566148 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":351,"threads_started":1}
I20260929 11:35:11.568379 17966 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46099
I20260929 11:35:11.568696 17966 heartbeater.cc:464] Registering TS with master...
I20260929 11:35:11.569398 17966 heartbeater.cc:511] Master 127.6.9.254:46099 requested a full tablet report, sending...
I20260929 11:35:11.570781 17475 ts_manager.cc:194] Registered new tserver with Master: 61f5b07d09cc456b96cf092f16c9aa6b (127.6.9.195:41917)
I20260929 11:35:11.571564 17475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:36657
I20260929 11:35:11.575889 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:35:11.592440 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.580230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.6.9.254:46099)
0929 11:35:11.580390 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.580394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.580864 (+ 470us) server_negotiation.cc:407] Connection header received
0929 11:35:11.581002 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.581006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.581079 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.581160 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:11.582585 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.583293 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.584643 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.585101 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.585188 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.587586 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:11.587590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:11.587594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:11.587625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:11.590202 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.590849 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.590853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.590855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.590967 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:11.591274 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:11.591277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:11.591279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:11.591522 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:11.591687 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.592060 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.592219 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:35:11.596493 17475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:35:11.603576 17475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:35:11.628571 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.615944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42207 (local address 127.0.0.1:57612)
0929 11:35:11.617120 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.617167 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.617337 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.618921 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.618930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.618965 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:35:11.619435 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.619447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.622665 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.622671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.625365 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.625387 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.627569 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.627577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.627837 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.627902 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.628030 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.628225 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":753,"threads_started":1}
I20260929 11:35:11.629261 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.616412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.6.9.194:42207)
0929 11:35:11.617873 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.617883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.617912 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:35:11.618016 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.618024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.618287 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.619676 (+ 1389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:35:11.619903 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.622485 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.625635 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.627288 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.628278 (+ 990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.628480 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.628553 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.628922 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":1195,"threads_started":1}
I20260929 11:35:11.633864 17767 tablet_service.cc:1467] Processing CreateTablet for tablet 9b50e6b8e930459a9871662446204856 (DEFAULT_TABLE table=test-table [id=5e70e795bdb04f5cac6444d8a66c5098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:35:11.635531 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b50e6b8e930459a9871662446204856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:35:11.636768 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.616114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:37513 (local address 127.0.0.1:58496)
0929 11:35:11.621115 (+ 5001us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.621138 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.621265 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.629633 (+ 8368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.629637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.629672 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:35:11.630434 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.630446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.632004 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.632010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.634228 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.634241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.636006 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.636011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.636134 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.636168 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.636253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.636401 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4864,"thread_start_us":4727,"threads_started":1}
I20260929 11:35:11.637121 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.616818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.6.9.193:37513)
0929 11:35:11.628294 (+ 11476us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.628305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.628341 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:35:11.628462 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.628473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.628808 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.629062 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:35:11.630630 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.631876 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.634427 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.635545 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.635674 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.635895 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.636329 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.636488 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11263,"thread_start_us":11094,"threads_started":1}
I20260929 11:35:11.640609 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.618077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:41917 (local address 127.0.0.1:33044)
0929 11:35:11.629264 (+ 11187us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.629282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.629473 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.630913 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.630918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.630944 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:35:11.631117 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.631127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.632996 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.633001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.637995 (+ 4994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.638021 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.639896 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.639902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.640083 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.640117 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.640203 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.640345 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11091,"thread_start_us":11317,"threads_started":1}
I20260929 11:35:11.640877 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.618660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.6.9.195:41917)
0929 11:35:11.630328 (+ 11668us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.630335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.630370 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:35:11.630459 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.630466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.630700 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.630908 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:35:11.631276 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.632824 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.638248 (+ 5424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.639708 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.639856 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.640089 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.640275 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.640441 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11510,"thread_start_us":11375,"threads_started":1}
I20260929 11:35:11.641717 17624 tablet_service.cc:1467] Processing CreateTablet for tablet 9b50e6b8e930459a9871662446204856 (DEFAULT_TABLE table=test-table [id=5e70e795bdb04f5cac6444d8a66c5098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:35:11.643240 17624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b50e6b8e930459a9871662446204856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:35:11.644783 17901 tablet_service.cc:1467] Processing CreateTablet for tablet 9b50e6b8e930459a9871662446204856 (DEFAULT_TABLE table=test-table [id=5e70e795bdb04f5cac6444d8a66c5098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:35:11.646238 17901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b50e6b8e930459a9871662446204856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:35:11.648893 17985 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Bootstrap starting.
I20260929 11:35:11.652585 17985 tablet_bootstrap.cc:654] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Neither blocks nor log segments found. Creating new log.
I20260929 11:35:11.653667 17985 log.cc:824] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:35:11.658571 17985 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: No bootstrap required, opened a new log
I20260929 11:35:11.658980 17985 ts_tablet_manager.cc:1397] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 11:35:11.660356 17987 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Bootstrap starting.
I20260929 11:35:11.662011 17988 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Bootstrap starting.
I20260929 11:35:11.664094 17987 tablet_bootstrap.cc:654] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Neither blocks nor log segments found. Creating new log.
I20260929 11:35:11.665077 17988 tablet_bootstrap.cc:654] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Neither blocks nor log segments found. Creating new log.
I20260929 11:35:11.665400 17987 log.cc:824] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Log is configured to *not* fsync() on all Append() calls
I20260929 11:35:11.666028 17988 log.cc:824] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:35:11.668494 17987 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: No bootstrap required, opened a new log
I20260929 11:35:11.668831 17987 ts_tablet_manager.cc:1397] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:35:11.669128 17985 raft_consensus.cc:348] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.669590 17985 raft_consensus.cc:374] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:35:11.669689 17985 raft_consensus.cc:729] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72160a8b5e8a40d4b02114935f7941b9, State: Initialized, Role: FOLLOWER
I20260929 11:35:11.669893 17988 tablet_bootstrap.cc:492] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: No bootstrap required, opened a new log
I20260929 11:35:11.670150 17988 ts_tablet_manager.cc:1397] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:35:11.670531 17985 consensus_queue.cc:260] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.671626 17832 heartbeater.cc:503] Master 127.6.9.254:46099 was elected leader, sending a full tablet report...
I20260929 11:35:11.672292 17985 ts_tablet_manager.cc:1428] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 11:35:11.677119 17987 raft_consensus.cc:348] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.677551 17987 raft_consensus.cc:374] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:35:11.677655 17987 raft_consensus.cc:729] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96b1ebe82d64e9785845f123b50df43, State: Initialized, Role: FOLLOWER
I20260929 11:35:11.678285 17987 consensus_queue.cc:260] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.679364 17988 raft_consensus.cc:348] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.679787 17988 raft_consensus.cc:374] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:35:11.679891 17988 raft_consensus.cc:729] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f5b07d09cc456b96cf092f16c9aa6b, State: Initialized, Role: FOLLOWER
I20260929 11:35:11.680028 17987 ts_tablet_manager.cc:1428] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 11:35:11.680465 17988 consensus_queue.cc:260] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.681543 17966 heartbeater.cc:503] Master 127.6.9.254:46099 was elected leader, sending a full tablet report...
I20260929 11:35:11.682252 17988 ts_tablet_manager.cc:1428] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
W20260929 11:35:11.690590 17690 tablet.cc:2369] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:35:11.758181 17833 tablet.cc:2369] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:35:11.798421 17967 tablet.cc:2369] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:35:11.957275 17991 raft_consensus.cc:482] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:35:11.957587 17991 raft_consensus.cc:504] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.959266 17991 leader_election.cc:288] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d96b1ebe82d64e9785845f123b50df43 (127.6.9.193:37513), 61f5b07d09cc456b96cf092f16c9aa6b (127.6.9.195:41917)
I20260929 11:35:11.966161 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.959591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:41917 (local address 127.6.9.194:37377)
0929 11:35:11.960267 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.960300 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.960517 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.960984 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.960990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.961020 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:35:11.961402 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.961413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.962495 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.962501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.964065 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.964077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.965399 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.965406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.965575 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.965610 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.965699 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.965849 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260929 11:35:11.966194 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.959692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:37377 (local address 127.6.9.195:41917)
0929 11:35:11.959880 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.959886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.960401 (+ 515us) server_negotiation.cc:407] Connection header received
0929 11:35:11.960723 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.960729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.960829 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.960944 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:35:11.961566 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.962371 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.964234 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.965180 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.965279 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.965359 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.965768 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.965910 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:35:11.966596 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.959789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:51817 (local address 127.6.9.193:37513)
0929 11:35:11.959958 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:11.959962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:11.960955 (+ 993us) server_negotiation.cc:407] Connection header received
0929 11:35:11.961172 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:11.961177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:11.961237 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:11.961312 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:35:11.961944 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.962796 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.964612 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.965533 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.965684 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.965782 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:11.966232 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 11:35:11.966380 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:35:11.967024 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:11.959591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:37513 (local address 127.6.9.194:51817)
0929 11:35:11.960846 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0929 11:35:11.960864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:35:11.960979 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:35:11.961353 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:35:11.961358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:35:11.961397 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:35:11.961745 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.961757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.962962 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.962966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.964426 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:35:11.964439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:35:11.965803 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:35:11.965810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:35:11.966024 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:11.966062 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:35:11.966200 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:35:11.966686 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":55,"thread_start_us":873,"threads_started":1}
I20260929 11:35:11.967322 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b50e6b8e930459a9871662446204856" candidate_uuid: "72160a8b5e8a40d4b02114935f7941b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" is_pre_election: true
I20260929 11:35:11.967787 17921 raft_consensus.cc:2454] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72160a8b5e8a40d4b02114935f7941b9 in term 0.
I20260929 11:35:11.967998 17644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b50e6b8e930459a9871662446204856" candidate_uuid: "72160a8b5e8a40d4b02114935f7941b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96b1ebe82d64e9785845f123b50df43" is_pre_election: true
I20260929 11:35:11.968454 17714 leader_election.cc:302] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f5b07d09cc456b96cf092f16c9aa6b, 72160a8b5e8a40d4b02114935f7941b9; no voters:
I20260929 11:35:11.968542 17644 raft_consensus.cc:2454] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72160a8b5e8a40d4b02114935f7941b9 in term 0.
I20260929 11:35:11.968935 17991 raft_consensus.cc:2781] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:35:11.969074 17991 raft_consensus.cc:482] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:35:11.969177 17991 raft_consensus.cc:3037] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:35:11.971410 17991 raft_consensus.cc:504] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.971992 17991 leader_election.cc:288] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [CANDIDATE]: Term 1 election: Requested vote from peers d96b1ebe82d64e9785845f123b50df43 (127.6.9.193:37513), 61f5b07d09cc456b96cf092f16c9aa6b (127.6.9.195:41917)
I20260929 11:35:11.972695 17644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b50e6b8e930459a9871662446204856" candidate_uuid: "72160a8b5e8a40d4b02114935f7941b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d96b1ebe82d64e9785845f123b50df43"
I20260929 11:35:11.972694 17921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b50e6b8e930459a9871662446204856" candidate_uuid: "72160a8b5e8a40d4b02114935f7941b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f5b07d09cc456b96cf092f16c9aa6b"
I20260929 11:35:11.972916 17921 raft_consensus.cc:3037] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:35:11.972916 17644 raft_consensus.cc:3037] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:35:11.976048 17921 raft_consensus.cc:2454] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72160a8b5e8a40d4b02114935f7941b9 in term 1.
I20260929 11:35:11.976048 17644 raft_consensus.cc:2454] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72160a8b5e8a40d4b02114935f7941b9 in term 1.
I20260929 11:35:11.976569 17713 leader_election.cc:302] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72160a8b5e8a40d4b02114935f7941b9, d96b1ebe82d64e9785845f123b50df43; no voters:
I20260929 11:35:11.976936 17991 raft_consensus.cc:2781] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:35:11.977375 17991 raft_consensus.cc:686] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 1 LEADER]: Becoming Leader. State: Replica: 72160a8b5e8a40d4b02114935f7941b9, State: Running, Role: LEADER
I20260929 11:35:11.977887 17991 consensus_queue.cc:237] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } }
I20260929 11:35:11.982975 17474 catalog_manager.cc:5495] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72160a8b5e8a40d4b02114935f7941b9 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "72160a8b5e8a40d4b02114935f7941b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72160a8b5e8a40d4b02114935f7941b9" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20260929 11:35:12.008371 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:12.002259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.6.9.194:42207)
0929 11:35:12.002483 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:12.002487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:12.002505 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:35:12.002638 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:12.002641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:12.002720 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:12.002852 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:35:12.003139 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.003848 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.005958 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.006436 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.006532 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:12.007176 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:35:12.007699 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:35:12.007833 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:12.007996 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 11:35:12.008074 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:35:12.017863 17644 raft_consensus.cc:1261] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 1 FOLLOWER]: Refusing update from remote peer 72160a8b5e8a40d4b02114935f7941b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:35:12.017889 17921 raft_consensus.cc:1261] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 1 FOLLOWER]: Refusing update from remote peer 72160a8b5e8a40d4b02114935f7941b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:35:12.018831 17996 consensus_queue.cc:1035] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f5b07d09cc456b96cf092f16c9aa6b" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:35:12.019308 17991 consensus_queue.cc:1035] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d96b1ebe82d64e9785845f123b50df43" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:35:12.032912 17999 mvcc.cc:205] Tried to move back new op lower bound from 7334632292413526016 to 7334632292265582592. Current Snapshot: MvccSnapshot[applied={T|T < 7334632292413526016}]
I20260929 11:35:12.033814 18000 mvcc.cc:205] Tried to move back new op lower bound from 7334632292413526016 to 7334632292265582592. Current Snapshot: MvccSnapshot[applied={T|T < 7334632292413526016}]
I20260929 11:35:12.045559 18001 mvcc.cc:205] Tried to move back new op lower bound from 7334632292413526016 to 7334632292265582592. Current Snapshot: MvccSnapshot[applied={T|T < 7334632292413526016}]
I20260929 11:35:12.055037 17474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:35:12.055296 17474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:35:12.058892 17474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 935d46617c5d409991fe51c9d02b310d: Sending DeleteTablet for 3 replicas of tablet 9b50e6b8e930459a9871662446204856
I20260929 11:35:12.060163 17767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b50e6b8e930459a9871662446204856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:35:12 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:57612
I20260929 11:35:12.060158 17624 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b50e6b8e930459a9871662446204856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:35:12 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:58496
I20260929 11:35:12.060834 17901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b50e6b8e930459a9871662446204856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:35:12 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:33044
I20260929 11:35:12.063239 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 17556
I20260929 11:35:12.073139 18011 tablet_replica.cc:330] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: stopping tablet replica
I20260929 11:35:12.073699 18011 raft_consensus.cc:2229] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:35:12.074246 18011 raft_consensus.cc:2258] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:35:12.076792 18011 ts_tablet_manager.cc:1905] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:35:12.085237 18011 ts_tablet_manager.cc:1918] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:35:12.085459 18011 log.cc:1196] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/9b50e6b8e930459a9871662446204856
I20260929 11:35:12.085994 18011 ts_tablet_manager.cc:1939] T 9b50e6b8e930459a9871662446204856 P 61f5b07d09cc456b96cf092f16c9aa6b: Deleting consensus metadata
I20260929 11:35:12.088135 17460 catalog_manager.cc:4841] TS 61f5b07d09cc456b96cf092f16c9aa6b (127.6.9.195:41917): tablet 9b50e6b8e930459a9871662446204856 (table test-table [id=5e70e795bdb04f5cac6444d8a66c5098]) successfully deleted
I20260929 11:35:12.091593 18010 tablet_replica.cc:330] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: stopping tablet replica
I20260929 11:35:12.092299 18010 raft_consensus.cc:2229] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:35:12.092969 18010 raft_consensus.cc:2258] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:35:12.095532 18010 ts_tablet_manager.cc:1905] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:35:12.096206 18009 tablet_replica.cc:330] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: stopping tablet replica
I20260929 11:35:12.096707 18009 raft_consensus.cc:2229] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:35:12.097463 18009 raft_consensus.cc:2258] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 11:35:12 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:35:12.099552 18009 ts_tablet_manager.cc:1905] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:35:12.104092 18010 ts_tablet_manager.cc:1918] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:35:12.104302 18010 log.cc:1196] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/9b50e6b8e930459a9871662446204856
I20260929 11:35:12.104938 18010 ts_tablet_manager.cc:1939] T 9b50e6b8e930459a9871662446204856 P d96b1ebe82d64e9785845f123b50df43: Deleting consensus metadata
I20260929 11:35:12.107061 17459 catalog_manager.cc:4841] TS d96b1ebe82d64e9785845f123b50df43 (127.6.9.193:37513): tablet 9b50e6b8e930459a9871662446204856 (table test-table [id=5e70e795bdb04f5cac6444d8a66c5098]) successfully deleted
I20260929 11:35:12.107493 18009 ts_tablet_manager.cc:1918] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:35:12.107681 18009 log.cc:1196] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/9b50e6b8e930459a9871662446204856
I20260929 11:35:12.108127 18009 ts_tablet_manager.cc:1939] T 9b50e6b8e930459a9871662446204856 P 72160a8b5e8a40d4b02114935f7941b9: Deleting consensus metadata
I20260929 11:35:12.109617 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:12.065685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.6.9.193:37513)
0929 11:35:12.065899 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:12.065906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:12.087876 (+ 21970us) server_negotiation.cc:407] Connection header received
0929 11:35:12.089095 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:12.089102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:12.089280 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:12.089377 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:12.090776 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.091899 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.093864 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.094527 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.094636 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:12.101087 (+ 6451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:12.101107 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:12.101125 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:12.101184 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:12.105590 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.106737 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.106747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.106751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.106921 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.107517 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.107523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.107527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.108085 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:12.108419 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:12.108951 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 11:35:12.109208 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:35:12.110322 17684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:35:12.110448 17462 catalog_manager.cc:4841] TS 72160a8b5e8a40d4b02114935f7941b9 (127.6.9.194:42207): tablet 9b50e6b8e930459a9871662446204856 (table test-table [id=5e70e795bdb04f5cac6444d8a66c5098]) successfully deleted
I20260929 11:35:12.200331 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 17556
I20260929 11:35:12.214479 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 17695
Sep 29 11:35:12 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:35:12.231891 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:12.215230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.6.9.194:42207)
0929 11:35:12.215472 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:12.215488 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:12.215537 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:35:12.215606 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:12.215612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:12.215693 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:12.215812 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:12.217040 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.218229 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.219663 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.220344 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.220472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:12.224656 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:12.224703 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:12.224726 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:12.224776 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:12.228898 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.229698 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.229709 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.229713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.229919 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.230312 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.230317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.230321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.230772 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:12.231092 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:12.231266 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 11:35:12.231487 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 11:35:12.232643 17827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:35:12.344367 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 17695
I20260929 11:35:12.359129 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 17838
Sep 29 11:35:12 dist-test-slave-c0l9 krb5kdc[16999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:35:12.377132 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:12.359963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.6.9.195:41917)
0929 11:35:12.360158 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:12.360162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:12.360181 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:35:12.360328 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:12.360332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:12.360401 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:12.360482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:12.361920 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.362706 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.364110 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:12.364518 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:12.364622 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:12.369069 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:12.369085 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:12.369103 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:12.369145 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:12.372767 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.373770 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.373781 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.373785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.373966 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:12.374487 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:12.374494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:12.374498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:12.375044 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:12.375361 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:12.376542 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 11:35:12.376781 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:35:12.377996 17961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:35:12.465049 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 17838
I20260929 11:35:12.478101 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 17443
I20260929 11:35:12.479386 17505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:35:12.589921 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 17443
I20260929 11:35:12.612027 6183 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:35:12.948045 6183 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:35:12.948275 6183 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:35:13.284178 6183 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:35:13.284 UTC [17029] LOG: received smart shutdown request
2026-09-29 11:35:13.287 UTC [17029] LOG: background worker "logical replication launcher" (PID 17038) exited with exit code 1
2026-09-29 11:35:13.288 UTC [17033] LOG: shutting down
2026-09-29 11:35:13.444 UTC [17029] LOG: database system is shut down
2026-09-29T11:35:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50662 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:35:13 dist-test-slave-c0l9 krb5kdc[18025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:35:13 dist-test-slave-c0l9 krb5kdc[18025](info): set up 2 sockets
Sep 29 11:35:13 dist-test-slave-c0l9 krb5kdc[18025](info): commencing operation
krb5kdc: starting...
W20260929 11:35:14.572809 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:35:14 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:35:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangeradmin_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangeradmin_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangerlookup_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangerlookup_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
I20260929 11:35:14.879666 6183 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvWjXHB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/postgres -l logfile start
2026-09-29 11:35:17.433 UTC [18055] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:35:17.433 UTC [18055] LOG: listening on IPv4 address "127.6.9.212", port 54473
2026-09-29 11:35:17.436 UTC [18055] LOG: listening on Unix socket "/tmp/.s.PGSQL.54473"
2026-09-29 11:35:17.461 UTC [18058] LOG: database system was shut down at 2026-09-29 11:35:15 UTC
2026-09-29 11:35:17.465 UTC [18055] LOG: database system is ready to accept connections
I20260929 11:35:18.356732 6183 mini_postgres.cc:96] Postgres bound to 54473
2026-09-29 11:35:18.372 UTC [18066] FATAL: database "slave" does not exist
127.6.9.212:54473 - accepting connections
I20260929 11:35:18.373399 6183 mini_ranger.cc:161] Starting Ranger...
I20260929 11:35:18.398676 6183 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:35:18.905978 6183 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:35:18.906075 6183 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin
2026-09-29 11:35:19,203 [I] DB FLAVOR :POSTGRES
2026-09-29 11:35:19,203 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:35:19,204 [I] Checking connection..
2026-09-29 11:35:19,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:35:19,461 [I] Checking connection passed.
2026-09-29 11:35:19,461 [I] --------- Verifying version history table ---------
2026-09-29 11:35:19,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:35:19,742 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:35:19,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:35:20,020 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:35:20,020 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:35:20,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:35:20.266 UTC [18130] WARNING: there is no transaction in progress
2026-09-29 11:35:20,272 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:35:20,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:35:20,520 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:35:20,520 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:35:20,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:35:20,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:35:21,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-c0l9.c.gcp-upstream.internal','N') ;"
2026-09-29 11:35:21,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:35:21,521 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:35:21,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:35:21,784 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:35:21,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:35:22,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:35:22,313 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:35:22,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:35:23.322 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.338 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.355 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.372 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.388 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.410 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.468 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.475 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.489 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.500 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.511 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.526 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.537 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.544 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.551 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:23.558 UTC [18238] WARNING: there is no transaction in progress
2026-09-29 11:35:24,246 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:35:24,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:35:24,483 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:35:24,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:35:24,751 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:35:24,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:35:24,993 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:35:24,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:35:25,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:35:25,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-c0l9.c.gcp-upstream.internal';"
2026-09-29 11:35:25,743 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:35:25,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:35:25,990 [I] DB_PATCHES have already been applied
I20260929 11:35:25.995962 6183 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvWjXHB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:35:25.996050 6183 mini_ranger.cc:193] Using host: 127.6.9.212
I20260929 11:35:26.010139 6183 mini_ranger.cc:227] Ranger admin URL: http://127.6.9.212:45449
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:35:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:35:27.046808 6183 mini_ranger.cc:160] Time spent starting Ranger: real 8.673s user 0.001s sys 0.038s
Sep 29, 2026 11:35:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:35:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:35:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:35:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:35:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:35:49.607920 6183 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.9.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.9.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangerkms_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.9.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/rangerkms_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.9.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.9.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.9.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/HTTP_127.6.9.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:35:49 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681749, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:35:49.782588 6183 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:35:49.802789 6183 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:35:50.694680 6183 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:35:50.694787 6183 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms
I20260929 11:35:50.694828 6183 mini_ranger_kms.cc:227] Using postgres at 127.6.9.212:54473
2026-09-29 11:35:51,051 [I] DB FLAVOR :POSTGRES
2026-09-29 11:35:51,051 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:35:51,051 [I] Checking connection
2026-09-29 11:35:51,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:35:51,297 [I] connection success
2026-09-29 11:35:51,297 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:35:51,297 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:35:51,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:35:51,542 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:35:51,542 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:35:51,542 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:35:51,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.9.212:54473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:35:51,800 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:35:52.283185 6183 mini_ranger_kms.cc:310] Created kms service
I20260929 11:35:52.453003 6183 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:35:52.502604 6183 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:35:52 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681749, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:52.955232 6183 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:35:52.955549 6183 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvWjXHB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvWjXHB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:35:52.955583 6183 mini_ranger_kms.cc:240] Using host: 127.6.9.212
I20260929 11:35:52.969427 6183 mini_ranger_kms.cc:276] Ranger KMS PID: 18419
I20260929 11:35:52.969662 6183 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.9.212:51937
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 11:35:55.086830 6183 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.304s user 0.093s sys 0.282s
I20260929 11:35:55.087067 6183 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:35:55.087225 6183 mini_ranger_kms.cc:401] 127.6.9.212:51937/kms/v1/keys
Sep 29 11:35:55 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681755, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 11:35:55 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681755, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
Sep 29 11:35:56 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681755, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.9.212@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
Sep 29 11:35:58 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:35:58.536590 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33679
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33679
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:35:58.886178 18469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:35:58.886511 18469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:35:58.886641 18469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:35:58.894179 18469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:35:58.894318 18469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:35:58.894395 18469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:35:58.894466 18469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:35:58.894529 18469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:35:58.908258 18469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.9.212:51937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33679
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:35:58.909680 18469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:35:58.911041 18469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:35:58.925028 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:35:58.924985 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:58.925945 18469 server_base.cc:1034] running on GCE node
W20260929 11:35:58.926015 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:35:58.927120 18469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:35:58.929316 18469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:35:58.930603 18469 hybrid_clock.cc:648] HybridClock initialized: now 1790681758930551 us; error 48 us; skew 500 ppm
I20260929 11:35:58.931066 18469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:35:58 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:35:58.935366 18469 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:35:58.937371 18469 webserver.cc:460] Webserver started at http://127.6.9.254:35149/ using document root <none> and password file <none>
I20260929 11:35:58.938324 18469 fs_manager.cc:359] Metadata directory not provided
I20260929 11:35:58.938474 18469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:35:58.938880 18469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:35:58 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:35:59.000339 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "93ab7009044448dbb720a466d75fb1fd"
format_stamp: "Formatted at 2026-09-29 11:35:58 on dist-test-slave-c0l9"
server_key: "e7ded595be46f6b77bfa8180010a831e"
server_key_iv: "7361ce39a56e5e0dee32490a3b894f2f"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:59.001592 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "93ab7009044448dbb720a466d75fb1fd"
format_stamp: "Formatted at 2026-09-29 11:35:58 on dist-test-slave-c0l9"
server_key: "e7ded595be46f6b77bfa8180010a831e"
server_key_iv: "7361ce39a56e5e0dee32490a3b894f2f"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:59.008948 18469 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 11:35:59.029966 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:35:59.031776 18469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:35:59.031985 18469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "93ab7009044448dbb720a466d75fb1fd"
format_stamp: "Formatted at 2026-09-29 11:35:58 on dist-test-slave-c0l9"
server_key: "e7ded595be46f6b77bfa8180010a831e"
server_key_iv: "7361ce39a56e5e0dee32490a3b894f2f"
server_key_version: "kuduclusterkey@0"
I20260929 11:35:59.032250 18469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:35:59.059015 18469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:35:59.064393 18469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:35:59.064777 18469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:35:59.084308 18535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33679 every 8 connection(s)
I20260929 11:35:59.084318 18469 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33679
I20260929 11:35:59.086259 18469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:35:59.090466 18536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:35:59.095822 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 18469
I20260929 11:35:59.096038 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:35:59.096879 6183 external_mini_cluster.cc:1428] Setting key cdf4ffbf946cdc9d51d0abaa2b20a934
I20260929 11:35:59.103827 18536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Bootstrap starting.
I20260929 11:35:59.108744 18536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Neither blocks nor log segments found. Creating new log.
I20260929 11:35:59.110492 18536 log.cc:824] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:35:59.117570 18536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: No bootstrap required, opened a new log
I20260929 11:35:59.127240 18536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:35:59.127784 18536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:35:59.127907 18536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ab7009044448dbb720a466d75fb1fd, State: Initialized, Role: FOLLOWER
I20260929 11:35:59.128647 18536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:35:59.128934 18536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:35:59.129084 18536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 29 11:35:59 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:35:59.129266 18536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:35:59.132047 18536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:35:59.132756 18536 leader_election.cc:302] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ab7009044448dbb720a466d75fb1fd; no voters:
I20260929 11:35:59.133487 18536 leader_election.cc:288] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:35:59.133630 18541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:35:59.142606 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:35:59.099190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.6.9.254:33679)
0929 11:35:59.114476 (+ 15286us) server_negotiation.cc:206] Beginning negotiation
0929 11:35:59.114494 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:35:59.114548 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:35:59.117085 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:35:59.117120 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:35:59.117697 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:35:59.118188 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:35:59.119882 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:59.122151 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:59.124081 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:35:59.125315 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:35:59.125525 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:35:59.130644 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:35:59.130657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:35:59.130686 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:35:59.130761 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:35:59.134916 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:59.136140 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:59.136152 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:59.136163 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:59.136399 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:35:59.136967 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:35:59.136973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:35:59.136976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:35:59.137985 (+ 1009us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:35:59.139973 (+ 1988us) server_negotiation.cc:1035] Waiting for connection context
0929 11:35:59.140341 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 11:35:59.140932 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15000,"thread_start_us":14746,"threads_started":1}
I20260929 11:35:59.145675 18541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 LEADER]: Becoming Leader. State: Replica: 93ab7009044448dbb720a466d75fb1fd, State: Running, Role: LEADER
I20260929 11:35:59.146358 18541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:35:59.147709 18536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:35:59.149224 18543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ab7009044448dbb720a466d75fb1fd. Latest consensus state: current_term: 1 leader_uuid: "93ab7009044448dbb720a466d75fb1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } } }
I20260929 11:35:59.150012 18543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: This master's current role is: LEADER
I20260929 11:35:59.149551 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ab7009044448dbb720a466d75fb1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } } }
I20260929 11:35:59.150614 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: This master's current role is: LEADER
I20260929 11:35:59.153188 18546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:35:59.158998 18546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:35:59.181648 18546 catalog_manager.cc:1347] Generated new cluster ID: d285c35e6555463898407bb67278f2b4
I20260929 11:35:59.181775 18546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:35:59.200723 18546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:35:59.202586 18546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:35:59.215862 18546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Generated new TSK 0
I20260929 11:35:59.217132 18546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:35:59.226621 18536 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:36:00.149334 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:00 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:36:00.640108 18581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:00.640607 18581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:00.640843 18581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:00.652793 18581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:00.653020 18581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:00.653249 18581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:00.669564 18581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:00.671604 18581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:00.673528 18581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:00.732025 18591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:00.733373 18592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:00.736765 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:00.737272 18581 server_base.cc:1034] running on GCE node
I20260929 11:36:00.738452 18581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:00.745044 18581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:00.746701 18581 hybrid_clock.cc:648] HybridClock initialized: now 1790681760746568 us; error 138 us; skew 500 ppm
I20260929 11:36:00.747231 18581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:00 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:00.751888 18581 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:00.772787 18581 webserver.cc:460] Webserver started at http://127.6.9.193:43639/ using document root <none> and password file <none>
I20260929 11:36:00.774070 18581 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:00.774338 18581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:00.774817 18581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:36:00 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:00.805675 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "09f185da05944a58b8b17e7f3ee2dd82"
format_stamp: "Formatted at 2026-09-29 11:36:00 on dist-test-slave-c0l9"
server_key: "17a8ddedb746d273d6ebd76bf3faab6e"
server_key_iv: "09bd99165e9f48d2d67b1a5e085add29"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:00.806615 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "09f185da05944a58b8b17e7f3ee2dd82"
format_stamp: "Formatted at 2026-09-29 11:36:00 on dist-test-slave-c0l9"
server_key: "17a8ddedb746d273d6ebd76bf3faab6e"
server_key_iv: "09bd99165e9f48d2d67b1a5e085add29"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:00.821159 18581 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260929 11:36:00.854223 18601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:00.856389 18581 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20260929 11:36:00.856758 18581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "09f185da05944a58b8b17e7f3ee2dd82"
format_stamp: "Formatted at 2026-09-29 11:36:00 on dist-test-slave-c0l9"
server_key: "17a8ddedb746d273d6ebd76bf3faab6e"
server_key_iv: "09bd99165e9f48d2d67b1a5e085add29"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:00.857129 18581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:00.935750 18581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:00.941329 18581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:00.941732 18581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:00.949642 18581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:00.965078 18581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:00.965279 18581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:00.965483 18581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:00.965823 18581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:36:01 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:01.305624 18581 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:35743
I20260929 11:36:01.305653 18713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:35743 every 8 connection(s)
I20260929 11:36:01.307622 18581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:01.313722 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 18581
I20260929 11:36:01.313974 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:36:01.314841 6183 external_mini_cluster.cc:1428] Setting key 3d82f7c79d6cf859fcc1fd41d9d08144
Sep 29 11:36:01 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:01.357404 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:01.311158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:50977 (local address 127.6.9.254:33679)
0929 11:36:01.321452 (+ 10294us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:01.321459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:01.326709 (+ 5250us) server_negotiation.cc:407] Connection header received
0929 11:36:01.328574 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:01.328582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:01.328731 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:01.328836 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:01.331534 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:01.332825 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:01.336184 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:01.336904 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:01.337035 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:01.348507 (+ 11472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:01.348516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:01.348521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:01.348578 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:01.352693 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:01.354021 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:01.354030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:01.354035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:01.354235 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:01.354955 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:01.354961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:01.354965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:01.355370 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:01.355505 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:01.356699 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0929 11:36:01.356989 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10135,"thread_start_us":9959,"threads_started":1}
I20260929 11:36:01.360186 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:01.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.193:50977)
0929 11:36:01.323717 (+ 11824us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:01.323808 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:01.328145 (+ 4337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:01.329503 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:01.329517 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:01.330271 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:01.331232 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:01.331266 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:01.333106 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:01.333113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:01.335872 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:01.335930 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:01.337459 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:01.337468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:01.337826 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:01.344561 (+ 6735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:01.344602 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:01.348094 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:01.353020 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:01.353033 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:01.353057 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:01.353815 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:01.354425 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:01.354431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:01.354435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:01.354799 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:01.355624 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:01.355634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:01.356164 (+ 530us) client_negotiation.cc:769] Sending connection context
0929 11:36:01.357958 (+ 1794us) client_negotiation.cc:240] Negotiation successful
0929 11:36:01.358609 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11411,"thread_start_us":14724,"threads_started":1}
I20260929 11:36:01.363647 18714 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:01.364032 18714 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:01.365048 18714 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:01.368010 18501 ts_manager.cc:194] Registered new tserver with Master: 09f185da05944a58b8b17e7f3ee2dd82 (127.6.9.193:35743)
I20260929 11:36:01.370314 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:50977
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:36:01.403936 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:01 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
W20260929 11:36:01.861615 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:01.861979 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:01.862107 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:01.869714 18720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:01.869854 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:01.870000 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:01.880364 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:01.881945 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:01.883411 18720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:01.898895 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:01.898912 18737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:01.900700 18720 server_base.cc:1034] running on GCE node
W20260929 11:36:01.901263 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:01.902143 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:01.903292 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:01.904551 18720 hybrid_clock.cc:648] HybridClock initialized: now 1790681761904483 us; error 57 us; skew 500 ppm
I20260929 11:36:01.904979 18720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:01 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:01.912142 18720 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:01.913864 18720 webserver.cc:460] Webserver started at http://127.6.9.194:45487/ using document root <none> and password file <none>
I20260929 11:36:01.914651 18720 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:01.914788 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:01.915086 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:36:01 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:01.932722 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "af39e7266ff24ede95ed4714ead325fb"
format_stamp: "Formatted at 2026-09-29 11:36:01 on dist-test-slave-c0l9"
server_key: "e300e3b1cc33e46d7da064e3f34ecc06"
server_key_iv: "d35af4cf6912c3893a29d42a4c3d3219"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:01.934101 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "af39e7266ff24ede95ed4714ead325fb"
format_stamp: "Formatted at 2026-09-29 11:36:01 on dist-test-slave-c0l9"
server_key: "e300e3b1cc33e46d7da064e3f34ecc06"
server_key_iv: "d35af4cf6912c3893a29d42a4c3d3219"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:01.941684 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 11:36:01.958815 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:01.960389 18720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:36:01.960587 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "af39e7266ff24ede95ed4714ead325fb"
format_stamp: "Formatted at 2026-09-29 11:36:01 on dist-test-slave-c0l9"
server_key: "e300e3b1cc33e46d7da064e3f34ecc06"
server_key_iv: "d35af4cf6912c3893a29d42a4c3d3219"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:01.960888 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:01.982806 18720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:01.987203 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:01.987509 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:01.988571 18720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:01.990278 18720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:01.990370 18720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:01.990492 18720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:01.990545 18720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:02.027204 18720 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:33133
I20260929 11:36:02.027204 18856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:33133 every 8 connection(s)
I20260929 11:36:02.028563 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:02.036973 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 18720
I20260929 11:36:02.037163 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:36:02.037767 6183 external_mini_cluster.cc:1428] Setting key c92ac99be619ce47578a4ec9d964e62c
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:36:02.079452 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.031633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:43969 (local address 127.6.9.254:33679)
0929 11:36:02.051074 (+ 19441us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:02.051080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:02.052345 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 11:36:02.054513 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:02.054520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:02.054622 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:02.054733 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:02.056967 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.058123 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.060761 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.061342 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.061485 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.066552 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:02.066561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:02.066567 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:02.066628 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:02.070574 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.076022 (+ 5448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.076030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.076033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.076257 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.076938 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.076943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.076946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.077369 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:02.077623 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:02.078791 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0929 11:36:02.079060 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19265,"thread_start_us":19432,"threads_started":1}
I20260929 11:36:02.082515 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.032359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.194:43969)
0929 11:36:02.051642 (+ 19283us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:02.051727 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:02.054134 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:02.055090 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:02.055105 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:02.055764 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:02.056718 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:02.056760 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.058350 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.058356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:02.060532 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:02.060552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.061620 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.061628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:02.061972 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.062856 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:02.062889 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:02.066190 (+ 3301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:02.074976 (+ 8786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:02.074990 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:02.075013 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:02.075768 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:02.076428 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:02.076433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:02.076437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:02.076774 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:02.077759 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:02.077769 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:02.078303 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 11:36:02.079721 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0929 11:36:02.080348 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18948,"thread_start_us":18694,"threads_started":1}
I20260929 11:36:02.085464 18857 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:02.085813 18857 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:02.086692 18857 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:02.088604 18501 ts_manager.cc:194] Registered new tserver with Master: af39e7266ff24ede95ed4714ead325fb (127.6.9.194:33133)
I20260929 11:36:02.089756 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:43969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:36:02.114166 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:02.375080 18714 heartbeater.cc:503] Master 127.6.9.254:33679 was elected leader, sending a full tablet report...
W20260929 11:36:02.456897 18863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:02.457195 18863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:02.457285 18863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:02.464993 18863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:02.465116 18863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:02.465231 18863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:02.477653 18863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:02.479239 18863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:02.480551 18863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:02.494565 18868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:02.494586 18869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:02.494529 18871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:02.495458 18863 server_base.cc:1034] running on GCE node
I20260929 11:36:02.496172 18863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:02.497324 18863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:02.498584 18863 hybrid_clock.cc:648] HybridClock initialized: now 1790681762498512 us; error 71 us; skew 500 ppm
I20260929 11:36:02.498946 18863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:02.502350 18863 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:02.503998 18863 webserver.cc:460] Webserver started at http://127.6.9.195:44517/ using document root <none> and password file <none>
I20260929 11:36:02.504806 18863 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:02.504925 18863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:02.505203 18863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:02.521862 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "ca192e8c08d043518454f55102d1d0f5"
format_stamp: "Formatted at 2026-09-29 11:36:02 on dist-test-slave-c0l9"
server_key: "12fd65d15c13a79250677c8cdcef2192"
server_key_iv: "98b96352bfe7e98703ce3cb95100cb7d"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:02.522833 18863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "ca192e8c08d043518454f55102d1d0f5"
format_stamp: "Formatted at 2026-09-29 11:36:02 on dist-test-slave-c0l9"
server_key: "12fd65d15c13a79250677c8cdcef2192"
server_key_iv: "98b96352bfe7e98703ce3cb95100cb7d"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:02.528954 18863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:36:02.547899 18878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:02.549572 18863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:36:02.549757 18863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "ca192e8c08d043518454f55102d1d0f5"
format_stamp: "Formatted at 2026-09-29 11:36:02 on dist-test-slave-c0l9"
server_key: "12fd65d15c13a79250677c8cdcef2192"
server_key_iv: "98b96352bfe7e98703ce3cb95100cb7d"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:02.549975 18863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:02.579681 18863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:02.583884 18863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:02.584189 18863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:02.585223 18863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:02.586669 18863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:02.586750 18863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:02.586872 18863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:02.586930 18863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:02.618486 18863 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:42783
I20260929 11:36:02.618510 18990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:42783 every 8 connection(s)
I20260929 11:36:02.619880 18863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:02.623519 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 18863
I20260929 11:36:02.623720 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:36:02.624326 6183 external_mini_cluster.cc:1428] Setting key 38d74ffb76398db87a4d56a6f6c50bb8
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681762, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:02.642138 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.622889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:50655 (local address 127.6.9.254:33679)
0929 11:36:02.623401 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:02.623407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:02.624505 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 11:36:02.626371 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:02.626378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:02.626479 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:02.626574 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:02.628228 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.629075 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.630825 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.631260 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.631386 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.635240 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:02.635246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:02.635250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:02.635303 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:02.638538 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.639512 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.639518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.639521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.639737 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.640318 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.640323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.640326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.640681 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:02.640847 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:02.641657 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 11:36:02.641884 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":294,"threads_started":1}
I20260929 11:36:02.643118 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.623355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.195:50655)
0929 11:36:02.624249 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:02.624296 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:02.626036 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:02.626874 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:02.626884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:02.627336 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:02.628007 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:02.628028 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.629260 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.629264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:02.630601 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:02.630626 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.631525 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.631533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:02.631805 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.632514 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:02.632540 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:02.634928 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:02.638726 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:02.638754 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:02.638770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:02.639362 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:02.639869 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:02.639874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:02.639878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:02.640156 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:02.640841 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:02.640849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:02.641276 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 11:36:02.641546 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 11:36:02.642001 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":416,"threads_started":1}
I20260929 11:36:02.644590 18991 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:02.644944 18991 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:02.645696 18991 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:02.647246 18501 ts_manager.cc:194] Registered new tserver with Master: ca192e8c08d043518454f55102d1d0f5 (127.6.9.195:42783)
I20260929 11:36:02.648083 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:50655
I20260929 11:36:02.653250 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:36:02.655058 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 18581
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:36:02.673485 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.656247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.6.9.193:35743)
0929 11:36:02.656867 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:02.656874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:02.656901 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:36:02.656958 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:02.656965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:02.657171 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:02.657311 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:02.658904 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.660125 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.661869 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.662545 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.662684 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.667507 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:02.667522 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:02.667539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:02.667580 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:02.670434 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.671306 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.671314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.671316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.671479 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.671931 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.671935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.671937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.672295 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:02.672531 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:02.672946 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 11:36:02.673184 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":296,"threads_started":1}
I20260929 11:36:02.674811 18709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:02.758047 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 18581
I20260929 11:36:02.770872 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 18720
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:36:02.788645 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.771795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.6.9.194:33133)
0929 11:36:02.772392 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:02.772400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:02.772430 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:36:02.772498 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:02.772504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:02.772767 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:02.772929 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:02.774291 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.775404 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.776939 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.777549 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.777665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.782680 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:02.782698 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:02.782714 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:02.782764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:02.785767 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.786618 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.786627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.786629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.786803 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.787146 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.787150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.787152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.787530 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:02.787777 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:02.788085 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 11:36:02.788339 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":318,"threads_started":1}
I20260929 11:36:02.789947 18852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:02.873695 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 18720
I20260929 11:36:02.886569 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 18863
Sep 29 11:36:02 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:36:02.902837 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:02.887551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.6.9.195:42783)
0929 11:36:02.888177 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:02.888184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:02.888219 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:36:02.888277 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:02.888283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:02.888491 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:02.888688 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:02.890086 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.891084 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.892345 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:02.892967 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:02.893079 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:02.897458 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:02.897473 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:02.897489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:02.897534 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:02.900060 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.900857 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.900867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.900871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.901012 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:02.901331 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:02.901333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:02.901335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:02.901664 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:02.901924 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:02.902261 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 11:36:02.902497 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":320,"threads_started":1}
I20260929 11:36:02.904165 18986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:02.986687 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 18863
I20260929 11:36:02.999576 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 18469
I20260929 11:36:03.000919 18531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:03.107781 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 18469
I20260929 11:36:03.126587 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33679
--webserver_interface=127.6.9.254
--webserver_port=35149
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33679
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:03.461285 19003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:03.461633 19003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:03.461774 19003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:03.469599 19003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:03.469748 19003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:03.469827 19003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:03.469916 19003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:03.469982 19003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:03.481014 19003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33679
--ranger_config_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=35149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:03.482335 19003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:03.483667 19003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:03.495733 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:03.495843 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:03.495796 19011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:03.496713 19003 server_base.cc:1034] running on GCE node
I20260929 11:36:03.497545 19003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:03.499097 19003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:03.500321 19003 hybrid_clock.cc:648] HybridClock initialized: now 1790681763500280 us; error 43 us; skew 500 ppm
I20260929 11:36:03.500727 19003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:03 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:03.504096 19003 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:03.505964 19003 webserver.cc:460] Webserver started at http://127.6.9.254:35149/ using document root <none> and password file <none>
I20260929 11:36:03.506705 19003 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:03.506844 19003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:36:03 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:03.578073 19003 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:36:03.582554 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:03.584331 19003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:36:03.584553 19003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "93ab7009044448dbb720a466d75fb1fd"
format_stamp: "Formatted at 2026-09-29 11:35:58 on dist-test-slave-c0l9"
server_key: "e7ded595be46f6b77bfa8180010a831e"
server_key_iv: "7361ce39a56e5e0dee32490a3b894f2f"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:03.585660 19003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:03.600216 19003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:03.604708 19003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:03.605023 19003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:03.623812 19003 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33679
I20260929 11:36:03.623844 19070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33679 every 8 connection(s)
I20260929 11:36:03.625424 19003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:03.631145 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19003
I20260929 11:36:03.631978 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:35743
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:33679
--webserver_port=43639
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:03.632902 19071 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:36:03.636330 19071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Bootstrap starting.
I20260929 11:36:03.655692 19071 log.cc:824] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:03.665343 19071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:36:03.665896 19071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Bootstrap complete.
I20260929 11:36:03.675137 19071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:36:03.676106 19071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ab7009044448dbb720a466d75fb1fd, State: Initialized, Role: FOLLOWER
I20260929 11:36:03.676862 19071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:36:03.677088 19071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:03.677206 19071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:03.677374 19071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:36:03.681622 19071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:36:03.682214 19071 leader_election.cc:302] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ab7009044448dbb720a466d75fb1fd; no voters:
I20260929 11:36:03.682865 19071 leader_election.cc:288] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:36:03.683037 19075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:36:03.684391 19075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [term 2 LEADER]: Becoming Leader. State: Replica: 93ab7009044448dbb720a466d75fb1fd, State: Running, Role: LEADER
I20260929 11:36:03.685081 19075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } }
I20260929 11:36:03.685444 19071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:03.687894 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93ab7009044448dbb720a466d75fb1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } } }
I20260929 11:36:03.688038 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ab7009044448dbb720a466d75fb1fd. Latest consensus state: current_term: 2 leader_uuid: "93ab7009044448dbb720a466d75fb1fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ab7009044448dbb720a466d75fb1fd" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33679 } } }
I20260929 11:36:03.688390 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:03.688289 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:03.703174 19082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:03.709477 19082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:03.710414 19071 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:36:03.711016 19082 catalog_manager.cc:1259] Loaded cluster ID: d285c35e6555463898407bb67278f2b4
I20260929 11:36:03.711118 19082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:03.715344 19082 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:03.717031 19082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Loaded TSK: 0
I20260929 11:36:03.718799 19082 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:36:04.107860 19073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:04.108218 19073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:04.108379 19073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:04.119261 19073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:04.119437 19073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:04.119633 19073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:04.134064 19073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:35743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=43639
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:04.135612 19073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:04.137485 19073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:04.165786 19108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:04.171005 19109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:04.178294 19111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:04.179014 19073 server_base.cc:1034] running on GCE node
I20260929 11:36:04.179895 19073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:04.187801 19073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:04.189376 19073 hybrid_clock.cc:648] HybridClock initialized: now 1790681764189260 us; error 103 us; skew 500 ppm
I20260929 11:36:04.189920 19073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:04 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:04.194890 19073 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:04.198580 19073 webserver.cc:460] Webserver started at http://127.6.9.193:43639/ using document root <none> and password file <none>
I20260929 11:36:04.199743 19073 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:04.199965 19073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:36:04 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:04.243068 19073 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.009s sys 0.001s
I20260929 11:36:04.248396 19119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:04.250816 19073 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 11:36:04.251180 19073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "09f185da05944a58b8b17e7f3ee2dd82"
format_stamp: "Formatted at 2026-09-29 11:36:00 on dist-test-slave-c0l9"
server_key: "17a8ddedb746d273d6ebd76bf3faab6e"
server_key_iv: "09bd99165e9f48d2d67b1a5e085add29"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:04.252703 19073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:04.280625 19073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:04.286734 19073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:04.287144 19073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:04.288414 19073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:04.290236 19073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:04.290349 19073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:04.290455 19073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:04.290521 19073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:04.359660 19073 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:35743
I20260929 11:36:04.359671 19232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:35743 every 8 connection(s)
I20260929 11:36:04.361580 19073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:04.369801 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19073
I20260929 11:36:04.370481 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:33133
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:33679
--webserver_port=45487
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:04 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:04.394598 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:04.365568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.193:54185)
0929 11:36:04.366605 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:04.366648 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:04.368589 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:04.371239 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:04.371252 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:04.371940 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:04.372857 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:04.372890 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:04.375635 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:04.375641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:04.377692 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:04.377718 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:04.379649 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:04.379657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:04.379962 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:04.380872 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:04.380904 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:04.383590 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:04.388684 (+ 5094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:04.388697 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:04.388719 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:04.389429 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:04.390070 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:04.390076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:04.390080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:04.390390 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:04.391507 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:04.391516 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:04.392012 (+ 496us) client_negotiation.cc:769] Sending connection context
0929 11:36:04.392362 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 11:36:04.393067 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":490,"threads_started":1}
I20260929 11:36:04.395450 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:04.365489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:54185 (local address 127.6.9.254:33679)
0929 11:36:04.366212 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:04.366233 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:04.366834 (+ 601us) server_negotiation.cc:407] Connection header received
0929 11:36:04.369453 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:04.369477 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:04.370103 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:04.370672 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:04.373186 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:04.375307 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:04.377982 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:04.379305 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:04.379583 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:04.383996 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:04.384008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:04.384032 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:04.384087 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:04.388387 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:04.389654 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:04.389666 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:04.389675 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:04.389916 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:04.390547 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:04.390553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:04.390556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:04.391334 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:04.391552 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:04.392721 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0929 11:36:04.393740 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260929 11:36:04.400133 19233 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:04.400525 19233 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:04.401503 19233 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
W20260929 11:36:04.405383 19233 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:33679 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:36:04.405622 19233 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:04.405925 19233 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:04.407521 19233 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:04.407923 19233 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
W20260929 11:36:04.409162 19233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:36:04.762379 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:04.762706 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:04.762879 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:04.770399 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:04.770535 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:04.770689 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:04.781708 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:33133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=45487
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:04.783172 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:04.784554 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:04.800750 19252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:04.800786 19250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:04.802279 19249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:04.802989 19237 server_base.cc:1034] running on GCE node
I20260929 11:36:04.803936 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:04.805341 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:04.806627 19237 hybrid_clock.cc:648] HybridClock initialized: now 1790681764806548 us; error 65 us; skew 500 ppm
I20260929 11:36:04.807073 19237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:04 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:04.811620 19237 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:04.813548 19237 webserver.cc:460] Webserver started at http://127.6.9.194:45487/ using document root <none> and password file <none>
I20260929 11:36:04.814313 19237 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:04.814474 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:36:04 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:04.851141 19237 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.009s sys 0.006s
I20260929 11:36:04.857108 19262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:04.859423 19237 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260929 11:36:04.859652 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "af39e7266ff24ede95ed4714ead325fb"
format_stamp: "Formatted at 2026-09-29 11:36:01 on dist-test-slave-c0l9"
server_key: "e300e3b1cc33e46d7da064e3f34ecc06"
server_key_iv: "d35af4cf6912c3893a29d42a4c3d3219"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:04.861241 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:04.886613 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:04.891095 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:04.891374 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:04.902422 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:04.910856 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:04.911015 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:04.911144 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:04.911221 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:05.014122 19237 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:33133
I20260929 11:36:05.014189 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:33133 every 8 connection(s)
I20260929 11:36:05.016104 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:05.017650 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19237
I20260929 11:36:05.018460 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.9.212:51937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:42783
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:33679
--webserver_port=44517
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:05.092737 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:05.026712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:57931 (local address 127.6.9.254:33679)
0929 11:36:05.060017 (+ 33305us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:05.060024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:05.064014 (+ 3990us) server_negotiation.cc:407] Connection header received
0929 11:36:05.064095 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:05.064100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:05.064196 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:05.064294 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:05.071012 (+ 6718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:05.072227 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:05.074881 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:05.075542 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:05.075669 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:05.081029 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:05.081039 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:05.081045 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:05.081095 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:05.084773 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:05.089168 (+ 4395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:05.089176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:05.089180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:05.089392 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:05.090148 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:05.090154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:05.090158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:05.090645 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:05.090802 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:05.092008 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0929 11:36:05.092285 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33149,"thread_start_us":33325,"threads_started":1}
I20260929 11:36:05.094355 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:05.026038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.194:57931)
0929 11:36:05.060778 (+ 34740us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:05.060856 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:05.063479 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:05.068330 (+ 4851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:05.068345 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:05.069081 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:05.069945 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:05.069972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:05.072489 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:05.072496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:05.074607 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:05.074630 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:05.076146 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:05.076156 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:05.076503 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:05.077371 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:05.077407 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:05.080599 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:05.088067 (+ 7468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:05.088079 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:05.088102 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:05.088933 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:05.089626 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:05.089632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:05.089636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:05.089974 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:05.090954 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:05.090975 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:05.091472 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 11:36:05.091844 (+ 372us) client_negotiation.cc:240] Negotiation successful
0929 11:36:05.092493 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34348,"thread_start_us":38254,"threads_started":1}
I20260929 11:36:05.096486 19377 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:05.096870 19377 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:05.097801 19377 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:05.100042 19036 ts_manager.cc:194] Registered new tserver with Master: af39e7266ff24ede95ed4714ead325fb (127.6.9.194:33133)
I20260929 11:36:05.102437 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:57931
I20260929 11:36:05.409309 19233 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:05.409801 19233 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:05.411098 19036 ts_manager.cc:194] Registered new tserver with Master: 09f185da05944a58b8b17e7f3ee2dd82 (127.6.9.193:35743)
I20260929 11:36:05.412289 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:54185
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
W20260929 11:36:05.489248 19380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:05.489677 19380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:05.489833 19380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:05.501485 19380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:05.501653 19380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:05.501834 19380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:05.517933 19380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.9.212:51937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:42783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=44517
--tserver_master_addrs=127.6.9.254:33679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:05.519711 19380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:05.521486 19380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:05.557701 19387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:05.562785 19388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:05.566748 19391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:05.567309 19380 server_base.cc:1034] running on GCE node
I20260929 11:36:05.568331 19380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:05.573290 19380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:05.576735 19380 hybrid_clock.cc:648] HybridClock initialized: now 1790681765576607 us; error 90 us; skew 500 ppm
I20260929 11:36:05.577278 19380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:05.583184 19380 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:05.602108 19380 webserver.cc:460] Webserver started at http://127.6.9.195:44517/ using document root <none> and password file <none>
I20260929 11:36:05.603549 19380 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:05.603777 19380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:05.666999 19380 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 11:36:05.685563 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:05.687934 19380 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260929 11:36:05.688232 19380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "ca192e8c08d043518454f55102d1d0f5"
format_stamp: "Formatted at 2026-09-29 11:36:02 on dist-test-slave-c0l9"
server_key: "12fd65d15c13a79250677c8cdcef2192"
server_key_iv: "98b96352bfe7e98703ce3cb95100cb7d"
server_key_version: "kuduclusterkey@0"
I20260929 11:36:05.689983 19380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:05.751551 19380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:05.757004 19380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:05.757347 19380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:05.764972 19380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:05.773483 19380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:05.773618 19380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:05.773725 19380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:05.773783 19380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:36:05 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for HTTP/127.6.9.212@KRBTEST.COM
I20260929 11:36:06.034662 19380 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:42783
I20260929 11:36:06.034708 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:42783 every 8 connection(s)
I20260929 11:36:06.036248 19380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:06.040720 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19380
I20260929 11:36:06.085093 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.042084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.6.9.254:33679)
0929 11:36:06.045795 (+ 3711us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.045804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.061805 (+ 16001us) server_negotiation.cc:407] Connection header received
0929 11:36:06.062029 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.062035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.062133 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.062266 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.066857 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.068088 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.071088 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.071845 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.072012 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.075879 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.075887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.075892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.075951 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.079993 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.081308 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.081315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.081319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.081551 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.083328 (+ 1777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.083334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.083337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.083723 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.084025 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.084391 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.084698 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3538,"thread_start_us":3337,"threads_started":1}
Sep 29 11:36:06 dist-test-slave-c0l9 krb5kdc[18025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:06.102218 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.065056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:36741 (local address 127.6.9.254:33679)
0929 11:36:06.066856 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.066863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.067764 (+ 901us) server_negotiation.cc:407] Connection header received
0929 11:36:06.076729 (+ 8965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.076735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.077181 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.077273 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.081442 (+ 4169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.082737 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.086755 (+ 4018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.087510 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.087612 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.093934 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.093944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.093949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.093999 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.097800 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.099130 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.099136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.099139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.099309 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.099911 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.099916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.099919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.100305 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.100464 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.101543 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.101820 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":3175,"threads_started":1}
I20260929 11:36:06.104931 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.066136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33679 (local address 127.6.9.195:36741)
0929 11:36:06.067585 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.067650 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.076294 (+ 8644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.078338 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.078365 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.079060 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:06.080002 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.080033 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.083643 (+ 3610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.083648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.086566 (+ 2918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.086587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.088026 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.088034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.088374 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.089218 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:06.089256 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:06.093512 (+ 4256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:06.098270 (+ 4758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:06.098283 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:06.098306 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:06.098980 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:06.099464 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:06.099470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:06.099473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:06.099766 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:06.100565 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:06.100574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:06.101090 (+ 516us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.102792 (+ 1702us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.103438 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":1916,"threads_started":1}
I20260929 11:36:06.107251 19512 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33679
I20260929 11:36:06.107373 19377 heartbeater.cc:503] Master 127.6.9.254:33679 was elected leader, sending a full tablet report...
I20260929 11:36:06.107632 19512 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:06.108590 19512 heartbeater.cc:511] Master 127.6.9.254:33679 requested a full tablet report, sending...
I20260929 11:36:06.110453 19036 ts_manager.cc:194] Registered new tserver with Master: ca192e8c08d043518454f55102d1d0f5 (127.6.9.195:42783)
I20260929 11:36:06.111697 19036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:36741
I20260929 11:36:06.114924 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:36:06.152402 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.127447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.6.9.254:33679)
0929 11:36:06.127639 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.127646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.133409 (+ 5763us) server_negotiation.cc:407] Connection header received
0929 11:36:06.133618 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.133624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.133744 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.133881 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.136036 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.137269 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.139356 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.140036 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.140188 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.143992 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.144001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.144006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.144061 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.147858 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.149009 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.149016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.149020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.149232 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.149754 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.149760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.149763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.150175 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.150428 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.151755 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.152041 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:36:06.159299 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:36:06.168869 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:36:06.199640 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.184989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:35743 (local address 127.0.0.1:43312)
0929 11:36:06.186556 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.186615 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.186837 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.190724 (+ 3887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.190733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.190779 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:06.191373 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.191392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.193637 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.193644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.196079 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.196101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.198538 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.198545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.198770 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.198832 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.198976 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.199216 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":6105,"threads_started":1}
I20260929 11:36:06.200830 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.185188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.6.9.193:35743)
0929 11:36:06.189846 (+ 4658us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.189854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.189891 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:36:06.190024 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.190034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.190414 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.190691 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:06.191671 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.193452 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.196320 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.197957 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.198106 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.198367 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.200256 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.200460 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4466,"thread_start_us":4261,"threads_started":1}
I20260929 11:36:06.202322 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.186305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.6.9.194:33133)
0929 11:36:06.189300 (+ 2995us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.189312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.191587 (+ 2275us) server_negotiation.cc:407] Connection header received
0929 11:36:06.191942 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.191952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.192233 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.192458 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:06.193470 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.195224 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.198542 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.200406 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.200517 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.200846 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.201672 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.201876 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2782,"thread_start_us":2572,"threads_started":1}
I20260929 11:36:06.203214 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.185692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:33133 (local address 127.0.0.1:35690)
0929 11:36:06.191432 (+ 5740us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.191460 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.191716 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.192836 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.192842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.192872 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:06.193243 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.193255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.195500 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.195507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.198317 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.198339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.201245 (+ 2906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.201251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.201432 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.201464 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.202694 (+ 1230us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.202848 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5585,"thread_start_us":802,"threads_started":1}
I20260929 11:36:06.207639 19312 tablet_service.cc:1467] Processing CreateTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 (DEFAULT_TABLE table=test-table [id=875a47e96ed24449876a4eeaa0e597dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:06.207643 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 (DEFAULT_TABLE table=test-table [id=875a47e96ed24449876a4eeaa0e597dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:06.209293 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78108cc775c54a78a4a1e78cd0649f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:06.209293 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78108cc775c54a78a4a1e78cd0649f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:06.210078 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.196873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42783 (local address 127.0.0.1:43938)
0929 11:36:06.199927 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.199947 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.200065 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.202591 (+ 2526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.202596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.202628 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:06.202813 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.202824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.205488 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.205494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.207307 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.207325 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.209181 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.209188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.209393 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.209433 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.209524 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.209787 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2940,"thread_start_us":5417,"threads_started":1}
I20260929 11:36:06.212814 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.192099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.6.9.195:42783)
0929 11:36:06.201548 (+ 9449us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.201558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.201589 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:36:06.201701 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.201709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.201998 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.203094 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:06.203434 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.205324 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.207512 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.208980 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.211881 (+ 2901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.212147 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.212248 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.212426 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9272,"thread_start_us":9120,"threads_started":1}
I20260929 11:36:06.217067 19447 tablet_service.cc:1467] Processing CreateTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 (DEFAULT_TABLE table=test-table [id=875a47e96ed24449876a4eeaa0e597dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:06.218716 19447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78108cc775c54a78a4a1e78cd0649f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:06.223325 19540 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Bootstrap starting.
I20260929 11:36:06.226897 19540 tablet_bootstrap.cc:654] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:06.228430 19540 log.cc:824] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:06.229012 19541 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Bootstrap starting.
I20260929 11:36:06.232436 19540 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: No bootstrap required, opened a new log
I20260929 11:36:06.232782 19540 ts_tablet_manager.cc:1397] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 11:36:06.232980 19541 tablet_bootstrap.cc:654] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:06.234102 19543 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Bootstrap starting.
I20260929 11:36:06.234936 19541 log.cc:824] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:06.237617 19543 tablet_bootstrap.cc:654] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:06.238695 19543 log.cc:824] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:06.239725 19541 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: No bootstrap required, opened a new log
I20260929 11:36:06.240053 19541 ts_tablet_manager.cc:1397] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 11:36:06.242272 19543 tablet_bootstrap.cc:492] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: No bootstrap required, opened a new log
I20260929 11:36:06.242574 19543 ts_tablet_manager.cc:1397] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260929 11:36:06.243005 19540 raft_consensus.cc:348] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.243494 19540 raft_consensus.cc:374] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:06.243592 19540 raft_consensus.cc:729] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af39e7266ff24ede95ed4714ead325fb, State: Initialized, Role: FOLLOWER
I20260929 11:36:06.244360 19540 consensus_queue.cc:260] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.246677 19540 ts_tablet_manager.cc:1428] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 11:36:06.251519 19541 raft_consensus.cc:348] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.252046 19541 raft_consensus.cc:374] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:06.252162 19541 raft_consensus.cc:729] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f185da05944a58b8b17e7f3ee2dd82, State: Initialized, Role: FOLLOWER
I20260929 11:36:06.252622 19543 raft_consensus.cc:348] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.253087 19543 raft_consensus.cc:374] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:06.252924 19541 consensus_queue.cc:260] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.253186 19543 raft_consensus.cc:729] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca192e8c08d043518454f55102d1d0f5, State: Initialized, Role: FOLLOWER
I20260929 11:36:06.253822 19543 consensus_queue.cc:260] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.254166 19233 heartbeater.cc:503] Master 127.6.9.254:33679 was elected leader, sending a full tablet report...
I20260929 11:36:06.254946 19541 ts_tablet_manager.cc:1428] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 11:36:06.255535 19543 ts_tablet_manager.cc:1428] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260929 11:36:06.255677 19512 heartbeater.cc:503] Master 127.6.9.254:33679 was elected leader, sending a full tablet report...
W20260929 11:36:06.323383 19378 tablet.cc:2369] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:36:06.333981 19515 tablet.cc:2369] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:36:06.364586 19234 tablet.cc:2369] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:06.430799 19547 raft_consensus.cc:482] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:06.431144 19547 raft_consensus.cc:504] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.432963 19547 leader_election.cc:288] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af39e7266ff24ede95ed4714ead325fb (127.6.9.194:33133), ca192e8c08d043518454f55102d1d0f5 (127.6.9.195:42783)
I20260929 11:36:06.439703 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.433622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58685 (local address 127.6.9.195:42783)
0929 11:36:06.433845 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.433851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.434497 (+ 646us) server_negotiation.cc:407] Connection header received
0929 11:36:06.434648 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.434651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.434709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.434791 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:06.435412 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.436104 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.437999 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.438783 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.438901 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.439013 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.439315 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.439454 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:36:06.439747 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.433477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:48131 (local address 127.6.9.194:33133)
0929 11:36:06.433708 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.433715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.434400 (+ 685us) server_negotiation.cc:407] Connection header received
0929 11:36:06.434580 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.434587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.434665 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.434734 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:06.435368 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.436093 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.438004 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.438776 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.438915 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.439017 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.439372 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.439503 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:36:06.439831 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.433141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:33133 (local address 127.6.9.193:48131)
0929 11:36:06.434278 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.434293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.434428 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.434771 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.434776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.434801 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:06.435211 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.435223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.436230 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.436234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.437844 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.437857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.439063 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.439068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.439197 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.439220 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.439322 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.439453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":105,"thread_start_us":810,"threads_started":1}
I20260929 11:36:06.440109 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.433467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42783 (local address 127.6.9.193:58685)
0929 11:36:06.434371 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:06.434389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:06.434529 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:06.434820 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:06.434824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:06.434846 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:06.435249 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.435257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.436225 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.436230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.437829 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:06.437843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.439019 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.439026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:06.439162 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.439188 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:36:06.439262 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 11:36:06.439402 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":22,"thread_start_us":570,"threads_started":1}
I20260929 11:36:06.441156 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78108cc775c54a78a4a1e78cd0649f61" candidate_uuid: "09f185da05944a58b8b17e7f3ee2dd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca192e8c08d043518454f55102d1d0f5" is_pre_election: true
I20260929 11:36:06.441155 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78108cc775c54a78a4a1e78cd0649f61" candidate_uuid: "09f185da05944a58b8b17e7f3ee2dd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af39e7266ff24ede95ed4714ead325fb" is_pre_election: true
I20260929 11:36:06.441653 19467 raft_consensus.cc:2454] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f185da05944a58b8b17e7f3ee2dd82 in term 0.
I20260929 11:36:06.441653 19332 raft_consensus.cc:2454] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09f185da05944a58b8b17e7f3ee2dd82 in term 0.
I20260929 11:36:06.442384 19120 leader_election.cc:302] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09f185da05944a58b8b17e7f3ee2dd82, af39e7266ff24ede95ed4714ead325fb, ca192e8c08d043518454f55102d1d0f5; no voters:
I20260929 11:36:06.442849 19547 raft_consensus.cc:2781] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:36:06.443017 19547 raft_consensus.cc:482] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:36:06.443111 19547 raft_consensus.cc:3037] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:06.445566 19547 raft_consensus.cc:504] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.446274 19547 leader_election.cc:288] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [CANDIDATE]: Term 1 election: Requested vote from peers af39e7266ff24ede95ed4714ead325fb (127.6.9.194:33133), ca192e8c08d043518454f55102d1d0f5 (127.6.9.195:42783)
I20260929 11:36:06.446923 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78108cc775c54a78a4a1e78cd0649f61" candidate_uuid: "09f185da05944a58b8b17e7f3ee2dd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca192e8c08d043518454f55102d1d0f5"
I20260929 11:36:06.446952 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78108cc775c54a78a4a1e78cd0649f61" candidate_uuid: "09f185da05944a58b8b17e7f3ee2dd82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af39e7266ff24ede95ed4714ead325fb"
I20260929 11:36:06.447126 19467 raft_consensus.cc:3037] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:06.447167 19332 raft_consensus.cc:3037] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:06.449481 19467 raft_consensus.cc:2454] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f185da05944a58b8b17e7f3ee2dd82 in term 1.
I20260929 11:36:06.449482 19332 raft_consensus.cc:2454] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09f185da05944a58b8b17e7f3ee2dd82 in term 1.
I20260929 11:36:06.450017 19122 leader_election.cc:302] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09f185da05944a58b8b17e7f3ee2dd82, af39e7266ff24ede95ed4714ead325fb, ca192e8c08d043518454f55102d1d0f5; no voters:
I20260929 11:36:06.450382 19547 raft_consensus.cc:2781] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:06.450878 19547 raft_consensus.cc:686] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 1 LEADER]: Becoming Leader. State: Replica: 09f185da05944a58b8b17e7f3ee2dd82, State: Running, Role: LEADER
I20260929 11:36:06.451438 19547 consensus_queue.cc:237] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } }
I20260929 11:36:06.456169 19036 catalog_manager.cc:5495] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09f185da05944a58b8b17e7f3ee2dd82 (127.6.9.193). New cstate: current_term: 1 leader_uuid: "09f185da05944a58b8b17e7f3ee2dd82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09f185da05944a58b8b17e7f3ee2dd82" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20260929 11:36:06.513211 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.508040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.6.9.193:35743)
0929 11:36:06.508254 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.508258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.508277 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:36:06.508467 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.508471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.508560 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.508733 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:36:06.509044 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.509801 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.511141 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.511535 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.511622 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.512103 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:36:06.512692 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:36:06.512817 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.512858 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.512948 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:36:06.522154 19467 raft_consensus.cc:1261] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 1 FOLLOWER]: Refusing update from remote peer 09f185da05944a58b8b17e7f3ee2dd82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:06.522157 19332 raft_consensus.cc:1261] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 1 FOLLOWER]: Refusing update from remote peer 09f185da05944a58b8b17e7f3ee2dd82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:06.523186 19551 consensus_queue.cc:1035] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af39e7266ff24ede95ed4714ead325fb" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:06.523558 19547 consensus_queue.cc:1035] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca192e8c08d043518454f55102d1d0f5" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:06.535432 19554 mvcc.cc:205] Tried to move back new op lower bound from 7334632515663921152 to 7334632515389009920. Current Snapshot: MvccSnapshot[applied={T|T < 7334632515663921152}]
I20260929 11:36:06.550056 19036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:06.550393 19036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:06.554603 19036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93ab7009044448dbb720a466d75fb1fd: Sending DeleteTablet for 3 replicas of tablet 78108cc775c54a78a4a1e78cd0649f61
I20260929 11:36:06.556082 19312 tablet_service.cc:1514] Processing DeleteTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:35690
I20260929 11:36:06.556125 19447 tablet_service.cc:1514] Processing DeleteTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:43938
I20260929 11:36:06.557257 19563 tablet_replica.cc:330] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: stopping tablet replica
I20260929 11:36:06.557785 19563 raft_consensus.cc:2229] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:06.558497 19563 raft_consensus.cc:2258] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:06.558588 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 78108cc775c54a78a4a1e78cd0649f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:43312
I20260929 11:36:06.559382 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19073
I20260929 11:36:06.559767 19564 tablet_replica.cc:330] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: stopping tablet replica
I20260929 11:36:06.560398 19564 raft_consensus.cc:2229] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:06.561156 19564 raft_consensus.cc:2258] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:06.562942 19565 tablet_replica.cc:330] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: stopping tablet replica
I20260929 11:36:06.563484 19565 raft_consensus.cc:2229] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:36:06.564219 19565 raft_consensus.cc:2258] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:06.565793 19563 ts_tablet_manager.cc:1905] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:06.566504 19565 ts_tablet_manager.cc:1905] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:06.573153 19563 ts_tablet_manager.cc:1918] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:06.574460 19565 ts_tablet_manager.cc:1918] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:06.574679 19565 log.cc:1196] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/78108cc775c54a78a4a1e78cd0649f61
I20260929 11:36:06.575315 19565 ts_tablet_manager.cc:1939] T 78108cc775c54a78a4a1e78cd0649f61 P 09f185da05944a58b8b17e7f3ee2dd82: Deleting consensus metadata
I20260929 11:36:06.575958 19564 ts_tablet_manager.cc:1905] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:06.577419 19020 catalog_manager.cc:4841] TS 09f185da05944a58b8b17e7f3ee2dd82 (127.6.9.193:35743): tablet 78108cc775c54a78a4a1e78cd0649f61 (table test-table [id=875a47e96ed24449876a4eeaa0e597dc]) successfully deleted
I20260929 11:36:06.578099 19563 log.cc:1196] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/78108cc775c54a78a4a1e78cd0649f61
I20260929 11:36:06.578682 19563 ts_tablet_manager.cc:1939] T 78108cc775c54a78a4a1e78cd0649f61 P ca192e8c08d043518454f55102d1d0f5: Deleting consensus metadata
I20260929 11:36:06.580690 19020 catalog_manager.cc:4841] TS ca192e8c08d043518454f55102d1d0f5 (127.6.9.195:42783): tablet 78108cc775c54a78a4a1e78cd0649f61 (table test-table [id=875a47e96ed24449876a4eeaa0e597dc]) successfully deleted
I20260929 11:36:06.583040 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.561260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.6.9.193:35743)
0929 11:36:06.561457 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.561463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.562183 (+ 720us) server_negotiation.cc:407] Connection header received
0929 11:36:06.562282 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.562287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.562396 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.562534 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.565470 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.566640 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.569620 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.570313 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.570446 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.574432 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.574453 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.574473 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.574538 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.579081 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.580181 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.580192 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.580195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.580416 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.580965 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.580970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.580973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.581375 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.581856 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.582390 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.582666 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:36:06.583774 19228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:06.584306 19564 ts_tablet_manager.cc:1918] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:06.584512 19564 log.cc:1196] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/78108cc775c54a78a4a1e78cd0649f61
I20260929 11:36:06.585139 19564 ts_tablet_manager.cc:1939] T 78108cc775c54a78a4a1e78cd0649f61 P af39e7266ff24ede95ed4714ead325fb: Deleting consensus metadata
I20260929 11:36:06.587412 19022 catalog_manager.cc:4841] TS af39e7266ff24ede95ed4714ead325fb (127.6.9.194:33133): tablet 78108cc775c54a78a4a1e78cd0649f61 (table test-table [id=875a47e96ed24449876a4eeaa0e597dc]) successfully deleted
I20260929 11:36:06.683971 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19073
I20260929 11:36:06.699198 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19237
I20260929 11:36:06.712575 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.699944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.6.9.194:33133)
0929 11:36:06.700102 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.700106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.700140 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:36:06.700324 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.700328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.700400 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.700487 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.701890 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.702703 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.704063 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.704478 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.704577 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.707091 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.707108 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.707125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.707166 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.709823 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.710517 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.710524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.710527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.710664 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.711039 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.711043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.711045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.711443 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.711744 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.712016 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.712227 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:36:06.713397 19372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:06.801298 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19237
I20260929 11:36:06.814321 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19380
I20260929 11:36:06.828483 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:06.815207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.6.9.195:42783)
0929 11:36:06.815704 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:06.815708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:06.815727 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:36:06.815786 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:06.815790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:06.815871 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:06.815967 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:06.817536 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.818260 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.819706 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:06.820145 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:06.820259 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:06.822950 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:06.822968 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:06.822984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:06.823022 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:06.825882 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.826680 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.826688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.826691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.826840 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:06.827167 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:06.827169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:06.827171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:06.827547 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:06.827775 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:06.827990 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 11:36:06.828156 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260929 11:36:06.829226 19507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:06.919001 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19380
I20260929 11:36:06.932709 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19003
I20260929 11:36:06.933960 19066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:07.045962 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19003
I20260929 11:36:07.065809 6183 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:36:07.401672 6183 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:36:07.401880 6183 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:36:07.739359 6183 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:36:07.739 UTC [18055] LOG: received smart shutdown request
2026-09-29 11:36:07.742 UTC [18055] LOG: background worker "logical replication launcher" (PID 18064) exited with exit code 1
2026-09-29 11:36:07.742 UTC [18059] LOG: shutting down
2026-09-29 11:36:07.914 UTC [18055] LOG: database system is shut down
2026-09-29T11:36:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54478 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:36:08 dist-test-slave-c0l9 krb5kdc[19578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:36:08 dist-test-slave-c0l9 krb5kdc[19578](info): set up 2 sockets
Sep 29 11:36:08 dist-test-slave-c0l9 krb5kdc[19578](info): commencing operation
krb5kdc: starting...
W20260929 11:36:09.049013 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:36:09 dist-test-slave-c0l9 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:36:09.280532 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46847
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:09.622677 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:09.623008 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:09.623134 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:09.630846 19591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:09.631021 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:09.631110 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:09.631193 19591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:09.631256 19591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:09.642230 19591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:09.643757 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:09.645252 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:09.658564 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:09.658726 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:09.658689 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:09.659221 19591 server_base.cc:1034] running on GCE node
I20260929 11:36:09.660072 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:09.661756 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:09.662983 19591 hybrid_clock.cc:648] HybridClock initialized: now 1790681769662937 us; error 47 us; skew 500 ppm
I20260929 11:36:09.663306 19591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:09 dist-test-slave-c0l9 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:09.666676 19591 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:09.668290 19591 webserver.cc:460] Webserver started at http://127.6.9.254:33553/ using document root <none> and password file <none>
I20260929 11:36:09.669094 19591 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:09.669220 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:09.669524 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:09.672655 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "7f730d9d6b81447a963ac530c227d551"
format_stamp: "Formatted at 2026-09-29 11:36:09 on dist-test-slave-c0l9"
server_key: "921f7fa6389ed561bdaf7f9704b0a5f5"
server_key_iv: "e149bdaf880215fa2b9c7fbe8349c4eb"
server_key_version: "encryptionkey@0"
I20260929 11:36:09.673566 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "7f730d9d6b81447a963ac530c227d551"
format_stamp: "Formatted at 2026-09-29 11:36:09 on dist-test-slave-c0l9"
server_key: "921f7fa6389ed561bdaf7f9704b0a5f5"
server_key_iv: "e149bdaf880215fa2b9c7fbe8349c4eb"
server_key_version: "encryptionkey@0"
I20260929 11:36:09.679157 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:36:09.683248 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:09.684901 19591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:09.685106 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7f730d9d6b81447a963ac530c227d551"
format_stamp: "Formatted at 2026-09-29 11:36:09 on dist-test-slave-c0l9"
server_key: "921f7fa6389ed561bdaf7f9704b0a5f5"
server_key_iv: "e149bdaf880215fa2b9c7fbe8349c4eb"
server_key_version: "encryptionkey@0"
I20260929 11:36:09.685318 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:09.716996 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:09.721359 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:09.721663 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:09.740206 19591 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46847
I20260929 11:36:09.740250 19657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46847 every 8 connection(s)
I20260929 11:36:09.741627 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:09.743544 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19591
I20260929 11:36:09.743778 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:36:09.744566 6183 external_mini_cluster.cc:1428] Setting key b835558c12b4ff4b978555bd2e9a8fdf
I20260929 11:36:09.746536 19658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:09.755295 19658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.9.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.253@KRBTEST.COM" created.
Sep 29 11:36:09 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:09.779973 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.758706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.6.9.254:46847)
0929 11:36:09.760552 (+ 1846us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:09.760561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:09.760623 (+ 62us) server_negotiation.cc:407] Connection header received
0929 11:36:09.762373 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:09.762379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:09.762664 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:09.762851 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:09.764212 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:09.765538 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:09.767009 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:09.767599 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:09.767789 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:09.771614 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:09.771626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:09.771646 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:09.771688 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:09.775446 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:09.776441 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:09.776449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:09.776451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:09.776619 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:09.777098 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:09.777101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:09.777103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:09.777651 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:09.777806 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:09.778468 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 11:36:09.778783 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":1613,"spinlock_wait_cycles":20864,"thread_start_us":1377,"threads_started":1}
I20260929 11:36:09.780033 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.758706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46847 (local address 127.0.0.1:55456)
0929 11:36:09.759657 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.759718 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:09.761979 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:09.762903 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:09.762910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:09.763638 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:09.764011 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:09.764032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:09.765693 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:09.765697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:09.766844 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:09.766865 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:09.767842 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:09.767849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:09.768066 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:09.768799 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:09.768825 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:09.771292 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:09.775644 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:09.775658 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:09.775669 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:09.776283 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:09.776749 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:09.776755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:09.776758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:09.776992 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:09.777804 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:09.777812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:09.778075 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 11:36:09.778354 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 11:36:09.778957 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":104,"thread_start_us":416,"threads_started":1}
I20260929 11:36:09.783633 19658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:36:09.785060 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.784547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50308)
0929 11:36:09.784785 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.784909 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 11:36:09.785408 19608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
I20260929 11:36:09.787242 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.786825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50320)
0929 11:36:09.787009 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.787096 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 11:36:09.787482 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43373: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
I20260929 11:36:09.801381 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:43373
--webserver_interface=127.6.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:09.836844 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } attempt: 1
I20260929 11:36:09.838284 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.837786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50322)
0929 11:36:09.838032 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.838128 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 11:36:09.839351 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.838993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50334)
0929 11:36:09.839131 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.839207 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:36:09.839558 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43373: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
I20260929 11:36:09.907815 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } attempt: 2
I20260929 11:36:09.909086 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.908559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50340)
0929 11:36:09.908825 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.908921 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 11:36:09.910162 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:09.909784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50356)
0929 11:36:09.909919 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:09.910023 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:36:09.910367 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43373: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
I20260929 11:36:10.001638 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } attempt: 3
I20260929 11:36:10.002961 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.002444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50370)
0929 11:36:10.002692 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.002810 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 11:36:10.004175 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.003774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50378)
0929 11:36:10.003945 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.004021 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 11:36:10.004386 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43373: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
W20260929 11:36:10.139784 19665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:10.140112 19665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:10.140246 19665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 11:36:10.147639 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } attempt: 4
W20260929 11:36:10.147769 19665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:10.147869 19665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:10.147928 19665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:10.148021 19665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:10.148087 19665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:10.148958 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.148354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50380)
0929 11:36:10.148720 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.148818 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 11:36:10.150053 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.149615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50390)
0929 11:36:10.149803 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.149902 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 11:36:10.150270 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43373: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43373: connect: Connection refused (error 111)
I20260929 11:36:10.158764 19665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:10.160136 19665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:10.161525 19665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:10.173298 19671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:10.173384 19670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:10.173341 19673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:10.174615 19665 server_base.cc:1034] running on GCE node
I20260929 11:36:10.175450 19665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:10.176996 19665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:10.178253 19665 hybrid_clock.cc:648] HybridClock initialized: now 1790681770178198 us; error 50 us; skew 500 ppm
I20260929 11:36:10.178635 19665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:10.181881 19665 init.cc:377] Logged in from keytab as kudu/127.6.9.253@KRBTEST.COM (short username kudu)
I20260929 11:36:10.183611 19665 webserver.cc:460] Webserver started at http://127.6.9.253:40483/ using document root <none> and password file <none>
I20260929 11:36:10.184343 19665 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:10.184468 19665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:10.184818 19665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:10.187857 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data/instance:
uuid: "a9aff2144f0047979d70f955cc6dba41"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "7765a2ff866ffea737d46967a42d6f59"
server_key_iv: "ea971a1a2ac2491b3387414658758ac9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.188772 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal/instance:
uuid: "a9aff2144f0047979d70f955cc6dba41"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "7765a2ff866ffea737d46967a42d6f59"
server_key_iv: "ea971a1a2ac2491b3387414658758ac9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.194387 19665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 11:36:10.197986 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:10.199383 19665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:36:10.199564 19665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "a9aff2144f0047979d70f955cc6dba41"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "7765a2ff866ffea737d46967a42d6f59"
server_key_iv: "ea971a1a2ac2491b3387414658758ac9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.199756 19665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:10.249270 19665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:10.254101 19665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:10.254375 19665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:10.272269 19665 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.253:43373
I20260929 11:36:10.272317 19731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.253:43373 every 8 connection(s)
I20260929 11:36:10.273701 19665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
I20260929 11:36:10.274267 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19665
I20260929 11:36:10.274477 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-1/wal/instance
I20260929 11:36:10.275031 6183 external_mini_cluster.cc:1428] Setting key 5d4f88d5ac45d48d1dfe434d8e074573
I20260929 11:36:10.278009 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:10.286448 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.9.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.252@KRBTEST.COM" created.
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:10.308331 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.290003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.6.9.254:46847)
0929 11:36:10.290549 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.290556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.291999 (+ 1443us) server_negotiation.cc:407] Connection header received
0929 11:36:10.293772 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.293778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.293866 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.293963 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.295612 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.296356 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.298005 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.298407 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.298532 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.302114 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.302121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.302126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.302179 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.305178 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.306074 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.306078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.306081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.306205 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.306621 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.306623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.306625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.306902 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.307017 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.307879 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.308060 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":253,"threads_started":1}
I20260929 11:36:10.309774 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.290697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46847 (local address 127.0.0.1:55458)
0929 11:36:10.291553 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.291603 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.293468 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.294239 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.294248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.294775 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.295427 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.295449 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.296523 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.296527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.297863 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.297877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.298649 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.298657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.298873 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.299517 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.299541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.301845 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.305319 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.305328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.305346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.305940 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.306294 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.306298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.306302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.306531 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.307014 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.307023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.307495 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.307800 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.308338 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260929 11:36:10.311839 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
I20260929 11:36:10.328164 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.312720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50396)
0929 11:36:10.312949 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.312968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.313091 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.314222 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.314226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.314655 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.314826 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.314835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.316828 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.316833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.317673 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.317684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.318849 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.318853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.319023 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.319543 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.319561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.322041 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.325558 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.325564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.325568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.326010 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.326464 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.326468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.326471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.326615 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.327226 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.327236 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.327376 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.327508 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.327758 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:36:10.328208 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.312940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.6.9.253:43373)
0929 11:36:10.313696 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.313704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.313736 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:36:10.313790 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.313798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.314052 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.314208 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.314994 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.316644 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.317816 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.318697 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.318829 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.322278 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.322301 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.322317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.322357 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.325379 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.326182 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.326191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.326194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.326335 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.326709 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.326713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.326715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.327097 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.327289 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.327701 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.327902 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":414,"threads_started":1}
I20260929 11:36:10.328720 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:44041
--webserver_interface=127.6.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:10.330773 19732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:36:10.332059 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.331510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34850)
0929 11:36:10.331695 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.331847 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:36:10.332338 19683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.333988 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.333596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34852)
0929 11:36:10.333757 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.333852 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:36:10.334198 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.383564 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 1
I20260929 11:36:10.384763 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.384247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34862)
0929 11:36:10.384447 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.384552 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 11:36:10.385936 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.385534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34870)
0929 11:36:10.385696 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.385784 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 11:36:10.386152 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.449573 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } attempt: 5
I20260929 11:36:10.454367 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 2
I20260929 11:36:10.455574 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.455074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34880)
0929 11:36:10.455277 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.455393 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:36:10.456795 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.456332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34882)
0929 11:36:10.456515 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.456606 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:36:10.457135 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:36:10.464156 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.450560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.6.9.253:43373)
0929 11:36:10.450774 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.450780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.450801 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:36:10.450924 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.450939 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.451031 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.451115 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.452012 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.452777 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.454057 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.454685 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.454821 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.458699 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.458705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.458709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.458746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.461388 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.462190 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.462195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.462197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.462329 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.462742 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.462746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.462748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.463052 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.463194 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.463596 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.463822 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 11:36:10.464172 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.450417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:50398)
0929 11:36:10.450628 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.450649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.450789 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.451190 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.451195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.451652 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.451841 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.451852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.452976 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.452980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.453893 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.453909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.454843 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.454850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.455145 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.455890 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.455915 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.458465 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.461622 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.461628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.461632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.462052 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.462458 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.462464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.462468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.462628 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.463175 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.463180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.463346 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.463474 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.463821 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:36:10.465479 19658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:36:10.466681 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.466245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34892)
0929 11:36:10.466448 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.466531 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:36:10.467696 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.467310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34904)
0929 11:36:10.467488 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.467568 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 11:36:10.467921 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.519177 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 1
I20260929 11:36:10.520301 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.519846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34916)
0929 11:36:10.520043 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.520134 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 11:36:10.521446 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.521067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34932)
0929 11:36:10.521235 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.521314 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 11:36:10.521669 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.548418 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 3
I20260929 11:36:10.549672 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.549164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34936)
0929 11:36:10.549403 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.549498 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:36:10.550840 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.550378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34950)
0929 11:36:10.550633 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.550717 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260929 11:36:10.551090 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.589928 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 2
I20260929 11:36:10.591202 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.590610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34952)
0929 11:36:10.590859 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.590978 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 11:36:10.592374 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.591942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:34960)
0929 11:36:10.592134 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.592228 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 11:36:10.592653 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
W20260929 11:36:10.669895 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:10.670228 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:10.670352 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:10.678097 19740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:10.678234 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:10.678282 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:10.678345 19740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:10.678390 19740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:10.688616 19740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46847,127.6.9.253:43373,127.6.9.252:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:10.690064 19740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:10.691468 19740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:36:10.694353 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 4
I20260929 11:36:10.695430 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.694972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41780)
0929 11:36:10.695197 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.695281 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:36:10.696455 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.696136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41788)
0929 11:36:10.696275 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.696346 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:36:10.696692 19732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
I20260929 11:36:10.698900 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 3
I20260929 11:36:10.699944 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.699475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41790)
0929 11:36:10.699652 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.699769 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 11:36:10.700966 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.700543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41792)
0929 11:36:10.700778 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.700853 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 11:36:10.701200 19658 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:44041: Network error: Client connection negotiation failed: client connection to 127.6.9.252:44041: connect: Connection refused (error 111)
W20260929 11:36:10.704161 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:10.704120 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:10.704200 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:10.710723 19740 server_base.cc:1034] running on GCE node
I20260929 11:36:10.711576 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:10.713155 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:10.714401 19740 hybrid_clock.cc:648] HybridClock initialized: now 1790681770714344 us; error 43 us; skew 500 ppm
I20260929 11:36:10.714771 19740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:10.718120 19740 init.cc:377] Logged in from keytab as kudu/127.6.9.252@KRBTEST.COM (short username kudu)
I20260929 11:36:10.719882 19740 webserver.cc:460] Webserver started at http://127.6.9.252:35089/ using document root <none> and password file <none>
I20260929 11:36:10.720872 19740 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:10.721046 19740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:10.721429 19740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:10.725178 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data/instance:
uuid: "e975f0b2a98c494a9282bc12e4af7403"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "92bba354b222101d9ea5928f42bde662"
server_key_iv: "67da91c0b21a74ef2217f2dab3bd6be9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.726034 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal/instance:
uuid: "e975f0b2a98c494a9282bc12e4af7403"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "92bba354b222101d9ea5928f42bde662"
server_key_iv: "67da91c0b21a74ef2217f2dab3bd6be9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.731592 19740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:36:10.735291 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:10.736840 19740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:10.737007 19740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "e975f0b2a98c494a9282bc12e4af7403"
format_stamp: "Formatted at 2026-09-29 11:36:10 on dist-test-slave-c0l9"
server_key: "92bba354b222101d9ea5928f42bde662"
server_key_iv: "67da91c0b21a74ef2217f2dab3bd6be9"
server_key_version: "encryptionkey@0"
I20260929 11:36:10.737198 19740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:10.757392 19740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:10.762022 19740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:10.762319 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:10.780539 19740 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.252:44041
I20260929 11:36:10.780558 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.252:44041 every 8 connection(s)
I20260929 11:36:10.781952 19740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
I20260929 11:36:10.786414 19807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:10.791816 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19740
I20260929 11:36:10.792035 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681566518919-6183-0/minicluster-data/master-2/wal/instance
I20260929 11:36:10.792614 6183 external_mini_cluster.cc:1428] Setting key b891897e98083a37b48fb8a56897cc48
I20260929 11:36:10.794530 19807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:36:10.814010 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.797512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.6.9.254:46847)
0929 11:36:10.797718 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.797725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.798639 (+ 914us) server_negotiation.cc:407] Connection header received
0929 11:36:10.798786 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.798792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.798911 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.799032 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.800304 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.801063 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.802780 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.803280 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.803408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.807698 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.807704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.807708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.807748 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.810465 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.811414 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.811418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.811421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.811561 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.812060 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.812065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.812067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.812443 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.812603 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.813495 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.813698 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:36:10.815205 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.794826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.6.9.252:44041)
0929 11:36:10.795730 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.795744 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.795789 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:36:10.797742 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.797773 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.798355 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.798711 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.800313 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.801758 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.803273 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.804001 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.804155 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.808595 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.808604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.808618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.808655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.811823 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.812783 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.812791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.812794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.812948 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.813305 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.813309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.813313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.813755 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.814035 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.814500 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.814743 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":679,"thread_start_us":453,"threads_started":1}
I20260929 11:36:10.815243 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.797548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46847 (local address 127.0.0.1:55406)
0929 11:36:10.798420 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.798471 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.798673 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.799109 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.799116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.799708 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.800116 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.800136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.801219 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.801225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.802631 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.802644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.803506 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.803512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.803817 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.804598 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.804631 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.807414 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.810617 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.810628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.810641 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.811271 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.811674 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.811680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.811684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.811938 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.812572 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.812579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.813106 (+ 527us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.813395 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.813939 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":52,"thread_start_us":467,"threads_started":1}
I20260929 11:36:10.816825 19807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:10.832350 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.817643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.6.9.253:43373)
0929 11:36:10.817850 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.817855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.817878 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:36:10.818353 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.818359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.818470 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.818569 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.819559 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.820636 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.822382 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.823062 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.823199 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.826746 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.826752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.826756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.826796 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.829664 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.830496 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.830500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.830502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.830685 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.831114 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.831117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.831120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.831382 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.831508 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.831896 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.832077 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:36:10.832487 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.817611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43373 (local address 127.0.0.1:34074)
0929 11:36:10.817785 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.817805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.817909 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.818761 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.818766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.819189 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.819385 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.819408 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.820814 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.820818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.822223 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.822240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.823219 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.823223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.823384 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.823994 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.824014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.826485 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.829818 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.829825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.829829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.830371 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.830799 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.830804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.830808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.831008 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.831502 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.831507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.831678 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.831838 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.832147 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 11:36:10.833570 19807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:36:10.835835 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.818918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.6.9.254:46847)
0929 11:36:10.819112 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.819118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.819136 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:36:10.819188 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.819193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.819272 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.819371 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.821004 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.822162 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.823807 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.824264 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.824361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.829048 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.829053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.829057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.829094 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.832275 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.833201 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.833207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.833210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.833386 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.834340 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.834345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.834348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.834681 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.834971 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.835287 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.835528 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:36:10.846787 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.834423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.6.9.252:44041)
0929 11:36:10.834574 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.834579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.834593 (+ 14us) server_negotiation.cc:407] Connection header received
0929 11:36:10.834625 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.834628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.834676 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.834750 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.835411 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.836278 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.837743 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.838161 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.838268 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.841352 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.841358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.841363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.841407 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.844284 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.844919 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.844925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.844928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.845047 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.845378 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.845382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.845383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.845660 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.845788 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.846191 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.846553 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":53}
I20260929 11:36:10.846812 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.834201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41798)
0929 11:36:10.834364 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.834379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.834466 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.834780 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.834784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.835147 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.835286 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.835295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.836445 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.836451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.837592 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.837608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.838288 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.838294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.838479 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.839059 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.839075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.841177 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.844406 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.844412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.844416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.844819 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.845146 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.845151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.845155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.845276 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.845769 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.845773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.845918 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.846067 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.846409 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 11:36:10.851774 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: Bootstrap starting.
I20260929 11:36:10.852397 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.837661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.6.9.253:43373)
0929 11:36:10.837827 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.837833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.837867 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:36:10.838054 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.838060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.838128 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.838206 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.839544 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.840380 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.841963 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.842495 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.842664 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.847112 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.847118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.847122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.847158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.849781 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.850495 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.850501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.850505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.850638 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.851017 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.851023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.851026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.851299 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.851551 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.851853 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.852093 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:36:10.855553 19807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:10.856921 19807 log.cc:824] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:10.857507 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:36:10.859586 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: No bootstrap required, opened a new log
I20260929 11:36:10.871749 19807 raft_consensus.cc:348] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:10.872263 19807 raft_consensus.cc:374] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:10.872396 19807 raft_consensus.cc:729] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e975f0b2a98c494a9282bc12e4af7403, State: Initialized, Role: FOLLOWER
I20260929 11:36:10.873358 19807 consensus_queue.cc:260] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:10.876266 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:10.876740 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:10.877655 19807 sys_catalog.cc:564] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:10.878432 19658 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 4
I20260929 11:36:10.893401 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.862231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.6.9.252:44041)
0929 11:36:10.862548 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.862554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.866161 (+ 3607us) server_negotiation.cc:407] Connection header received
0929 11:36:10.866742 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.866749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.866848 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.866928 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.870289 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.871440 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.873903 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.874712 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.874858 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.885644 (+ 10786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.885652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.885656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.885707 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.889223 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.890251 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.890257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.890260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.890580 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.891125 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.891130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.891133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.891603 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.891845 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.892680 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.892946 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
Sep 29 11:36:10 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681769, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:36:10.900029 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.862181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.6.9.254:46847)
0929 11:36:10.862377 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.862383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.870290 (+ 7907us) server_negotiation.cc:407] Connection header received
0929 11:36:10.870444 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.870450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.870553 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.870645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.873942 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.875294 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.877608 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.878285 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.878461 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.883411 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.883418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.883423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.883468 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.893500 (+ 10032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.896786 (+ 3286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.896793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.896797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.896983 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.897540 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.897546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.897549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.898009 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.898360 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.899404 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.899661 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:36:10.900621 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.862210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.6.9.253:43373)
0929 11:36:10.862981 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.862987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.869999 (+ 7012us) server_negotiation.cc:407] Connection header received
0929 11:36:10.870288 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.870295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.870403 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.870543 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.873909 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.875021 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.878685 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.879380 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.879536 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.887967 (+ 8431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.887974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.887979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.888029 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.891665 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.896709 (+ 5044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.896717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.896721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.896954 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.897420 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.897425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.897427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.897739 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.898068 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.899928 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.900368 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260929 11:36:10.906564 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.879051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41810)
0929 11:36:10.879455 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.879473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.879580 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.881256 (+ 1676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.881262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.881799 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.882016 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.882030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.883864 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.883871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.885668 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.885702 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.886968 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.886975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:10.887180 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.893952 (+ 6772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:10.893978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:10.896991 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:10.903848 (+ 6857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.903854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.903857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.904473 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.905035 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:10.905039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:10.905042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:10.905189 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:10.905822 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:10.905825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:10.905965 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 11:36:10.906070 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:36:10.906293 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
I20260929 11:36:10.906903 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.879923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.6.9.252:44041)
0929 11:36:10.880859 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.880867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.880895 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:36:10.880966 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.880974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.881069 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.881190 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.882221 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.883664 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.885936 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.886763 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.898495 (+ 11732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:10.898795 (+ 300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:10.898802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:10.898808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:10.898841 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:10.903523 (+ 4682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.904706 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.904713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.904717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.904926 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:10.905282 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:10.905287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:10.905290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:10.905718 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:10.905865 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:10.906192 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 11:36:10.906455 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":531,"threads_started":1}
I20260929 11:36:10.912094 19658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: Bootstrap starting.
I20260929 11:36:10.916384 19658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:10.917793 19658 log.cc:824] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: Log is configured to *not* fsync() on all Append() calls
W20260929 11:36:10.921905 19832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 11:36:10.911345 19740 master_runner.cc:186] Error getting master registration for 127.6.9.254:46847: OK, instance_id { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" instance_seqno: 1790681769734903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260929 11:36:10.922037 19832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:36:10.922248 19740 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:36:10.927237 19658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: No bootstrap required, opened a new log
I20260929 11:36:10.935585 19658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:10.935949 19658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:10.936074 19658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f730d9d6b81447a963ac530c227d551, State: Initialized, Role: FOLLOWER
I20260929 11:36:10.936759 19658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:10.937750 19835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:10.938213 19835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:10.939299 19658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:36:10.952195 19846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:36:10.952363 19846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:36:10.949891 19591 master_runner.cc:186] Error getting master registration for 127.6.9.253:43373: OK, instance_id { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" instance_seqno: 1790681770267191 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:36:10.958169 19591 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 11:36:10.974757 19612 tablet.cc:2369] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:10.995956 19732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } attempt: 5
Sep 29 11:36:11 dist-test-slave-c0l9 krb5kdc[19578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:36:11.012933 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.996813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:44041 (local address 127.0.0.1:41814)
0929 11:36:10.997114 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:10.997134 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:10.997301 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:10.997720 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:10.997725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:10.998173 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:10.998367 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:10.998378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.999529 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:10.999534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:11.000551 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:11.000568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:11.003448 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:11.003455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:11.003691 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:11.004497 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:11.004518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:11.007015 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:11.010360 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:11.010367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:11.010371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:11.010787 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:11.011193 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:11.011199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:11.011202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:11.011347 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:11.011885 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:11.011890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:11.012042 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 11:36:11.012172 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 11:36:11.012495 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260929 11:36:11.012923 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:10.996891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.6.9.252:44041)
0929 11:36:10.997097 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:10.997104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:10.997221 (+ 117us) server_negotiation.cc:407] Connection header received
0929 11:36:10.997458 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:10.997462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:10.997561 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:10.997665 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:10.998569 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:10.999356 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:11.000754 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:11.001441 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:11.001575 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:11.007258 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:11.007264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:11.007267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:11.007305 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:11.010193 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:11.010925 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:11.010930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:11.010932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:11.011076 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:11.011454 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:11.011458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:11.011460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:11.011762 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:11.011901 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:11.012284 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 11:36:11.012528 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 11:36:11.014637 19761 tablet.cc:2369] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:11.018363 19732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41: Bootstrap starting.
I20260929 11:36:11.023765 19732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:11.024899 19732 log.cc:824] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:11.027701 19732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41: No bootstrap required, opened a new log
I20260929 11:36:11.035101 19732 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.035497 19732 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:11.035615 19732 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9aff2144f0047979d70f955cc6dba41, State: Initialized, Role: FOLLOWER
I20260929 11:36:11.036283 19732 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.039113 19849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.039618 19849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:11.041234 19732 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:11.045017 19849 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:11.045370 19849 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.047200 19849 leader_election.cc:288] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f730d9d6b81447a963ac530c227d551 (127.6.9.254:46847), e975f0b2a98c494a9282bc12e4af7403 (127.6.9.252:44041)
I20260929 11:36:11.048990 19835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:11.049283 19835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.049204 19782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9aff2144f0047979d70f955cc6dba41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e975f0b2a98c494a9282bc12e4af7403" is_pre_election: true
I20260929 11:36:11.049896 19782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9aff2144f0047979d70f955cc6dba41 in term 0.
I20260929 11:36:11.050925 19683 leader_election.cc:302] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9aff2144f0047979d70f955cc6dba41, e975f0b2a98c494a9282bc12e4af7403; no voters:
I20260929 11:36:11.051003 19835 leader_election.cc:288] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9aff2144f0047979d70f955cc6dba41 (127.6.9.253:43373), e975f0b2a98c494a9282bc12e4af7403 (127.6.9.252:44041)
I20260929 11:36:11.051541 19849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:36:11.051714 19849 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:36:11.051828 19849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:11.052259 19707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f730d9d6b81447a963ac530c227d551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9aff2144f0047979d70f955cc6dba41" is_pre_election: true
I20260929 11:36:11.055303 19849 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.055627 19633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9aff2144f0047979d70f955cc6dba41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f730d9d6b81447a963ac530c227d551" is_pre_election: true
I20260929 11:36:11.056236 19633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9aff2144f0047979d70f955cc6dba41 in term 0.
I20260929 11:36:11.056818 19849 leader_election.cc:288] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [CANDIDATE]: Term 1 election: Requested vote from peers 7f730d9d6b81447a963ac530c227d551 (127.6.9.254:46847), e975f0b2a98c494a9282bc12e4af7403 (127.6.9.252:44041)
I20260929 11:36:11.057263 19782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9aff2144f0047979d70f955cc6dba41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e975f0b2a98c494a9282bc12e4af7403"
I20260929 11:36:11.057776 19782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:11.058063 19781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f730d9d6b81447a963ac530c227d551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e975f0b2a98c494a9282bc12e4af7403" is_pre_election: true
I20260929 11:36:11.056886 19707 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f730d9d6b81447a963ac530c227d551 in current term 1: Already voted for candidate a9aff2144f0047979d70f955cc6dba41 in this term.
I20260929 11:36:11.060124 19608 leader_election.cc:302] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f730d9d6b81447a963ac530c227d551; no voters: a9aff2144f0047979d70f955cc6dba41, e975f0b2a98c494a9282bc12e4af7403
I20260929 11:36:11.060272 19633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a9aff2144f0047979d70f955cc6dba41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f730d9d6b81447a963ac530c227d551"
I20260929 11:36:11.060473 19633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:11.061362 19782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9aff2144f0047979d70f955cc6dba41 in term 1.
I20260929 11:36:11.062991 19683 leader_election.cc:302] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9aff2144f0047979d70f955cc6dba41, e975f0b2a98c494a9282bc12e4af7403; no voters:
I20260929 11:36:11.063408 19849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:11.063706 19633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9aff2144f0047979d70f955cc6dba41 in term 1.
I20260929 11:36:11.065306 19835 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 11:36:11.071777 19849 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [term 1 LEADER]: Becoming Leader. State: Replica: a9aff2144f0047979d70f955cc6dba41, State: Running, Role: LEADER
I20260929 11:36:11.072567 19849 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } }
I20260929 11:36:11.086468 19854 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9aff2144f0047979d70f955cc6dba41. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.086823 19854 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:11.087704 19859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:11.116756 19859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:11.140738 19782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [term 1 FOLLOWER]: Refusing update from remote peer a9aff2144f0047979d70f955cc6dba41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:11.141850 19854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:11.162446 19633 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [term 1 FOLLOWER]: Refusing update from remote peer a9aff2144f0047979d70f955cc6dba41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:11.163503 19854 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:11.171263 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9aff2144f0047979d70f955cc6dba41. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.171615 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:11.176466 19865 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:36:11.177169 19866 mvcc.cc:205] Tried to move back new op lower bound from 7334632534576148480 to 7334632534320726016. Current Snapshot: MvccSnapshot[applied={T|T < 7334632534576148480}]
I20260929 11:36:11.177192 19854 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.177742 19854 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:11.178728 19864 mvcc.cc:205] Tried to move back new op lower bound from 7334632534576148480 to 7334632534320726016. Current Snapshot: MvccSnapshot[applied={T|T < 7334632534576148480}]
I20260929 11:36:11.178938 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.179205 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:11.185506 19859 catalog_manager.cc:1347] Generated new cluster ID: b88d69a21d684212a54790dd3224fa05
I20260929 11:36:11.185626 19859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:11.214337 19835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9aff2144f0047979d70f955cc6dba41. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.214702 19835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:11.219058 19849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.219408 19849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:11.226678 19859 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:36:11.228864 19859 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:11.244514 19835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a9aff2144f0047979d70f955cc6dba41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f730d9d6b81447a963ac530c227d551" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46847 } } peers { permanent_uuid: "a9aff2144f0047979d70f955cc6dba41" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43373 } } peers { permanent_uuid: "e975f0b2a98c494a9282bc12e4af7403" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 44041 } } }
I20260929 11:36:11.244827 19835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:36:11.249966 19859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9aff2144f0047979d70f955cc6dba41: Generated new TSK 0
I20260929 11:36:11.251078 19859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:36:11.924115 19832 catalog_manager.cc:1259] Loaded cluster ID: b88d69a21d684212a54790dd3224fa05
I20260929 11:36:11.924225 19832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: loading cluster ID for follower catalog manager: success
I20260929 11:36:11.926733 19832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: acquiring CA information for follower catalog manager: success
I20260929 11:36:11.928571 19832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e975f0b2a98c494a9282bc12e4af7403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:36:11.954484 19846 catalog_manager.cc:1259] Loaded cluster ID: b88d69a21d684212a54790dd3224fa05
I20260929 11:36:11.954599 19846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: loading cluster ID for follower catalog manager: success
I20260929 11:36:11.957202 19846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: acquiring CA information for follower catalog manager: success
I20260929 11:36:11.959189 19846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f730d9d6b81447a963ac530c227d551: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:36:12.418717 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19591
I20260929 11:36:12.419910 19653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:12.532486 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19591
I20260929 11:36:12.549957 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19665
I20260929 11:36:12.551105 19727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:12.658263 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19665
I20260929 11:36:12.675361 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19740
I20260929 11:36:12.676631 19802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:12.784610 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19740
2026-09-29T11:36:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4760 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:36:12 dist-test-slave-c0l9 krb5kdc[19882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:36:12 dist-test-slave-c0l9 krb5kdc[19882](info): set up 2 sockets
Sep 29 11:36:12 dist-test-slave-c0l9 krb5kdc[19882](info): commencing operation
krb5kdc: starting...
W20260929 11:36:13.809224 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:36:13 dist-test-slave-c0l9 krb5kdc[19882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:36:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:36:14.041673 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40045
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:14.386361 19895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:14.386711 19895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:14.386852 19895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:14.394520 19895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:14.394676 19895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:14.394745 19895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:14.394827 19895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:14.394888 19895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:14.406037 19895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:14.407424 19895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:14.408895 19895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:14.421221 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:14.421154 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:14.421113 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:14.421622 19895 server_base.cc:1034] running on GCE node
I20260929 11:36:14.422988 19895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:14.424618 19895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:14.425902 19895 hybrid_clock.cc:648] HybridClock initialized: now 1790681774425839 us; error 48 us; skew 500 ppm
I20260929 11:36:14.426340 19895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:14 dist-test-slave-c0l9 krb5kdc[19882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681774, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:14.433872 19895 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:14.435631 19895 webserver.cc:460] Webserver started at http://127.6.9.254:44973/ using document root <none> and password file <none>
I20260929 11:36:14.436401 19895 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:14.436533 19895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:14.436896 19895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:14.440037 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "d2fd6910a4ea40a89f64a08d720f4181"
format_stamp: "Formatted at 2026-09-29 11:36:14 on dist-test-slave-c0l9"
server_key: "0445a831cb609b7ce2ff68adab22663c"
server_key_iv: "80fc0404049ebe2712c34b0ca4c0e8a6"
server_key_version: "encryptionkey@0"
I20260929 11:36:14.440871 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "d2fd6910a4ea40a89f64a08d720f4181"
format_stamp: "Formatted at 2026-09-29 11:36:14 on dist-test-slave-c0l9"
server_key: "0445a831cb609b7ce2ff68adab22663c"
server_key_iv: "80fc0404049ebe2712c34b0ca4c0e8a6"
server_key_version: "encryptionkey@0"
I20260929 11:36:14.446475 19895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:36:14.450237 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:14.451669 19895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260929 11:36:14.451858 19895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "d2fd6910a4ea40a89f64a08d720f4181"
format_stamp: "Formatted at 2026-09-29 11:36:14 on dist-test-slave-c0l9"
server_key: "0445a831cb609b7ce2ff68adab22663c"
server_key_iv: "80fc0404049ebe2712c34b0ca4c0e8a6"
server_key_version: "encryptionkey@0"
I20260929 11:36:14.452062 19895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:14.478727 19895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:14.483584 19895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:14.483906 19895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:14.504563 19895 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40045
I20260929 11:36:14.504617 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40045 every 8 connection(s)
I20260929 11:36:14.506060 19895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:14.510288 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:14.516213 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 19895
I20260929 11:36:14.516477 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:36:14.517437 6183 external_mini_cluster.cc:1428] Setting key 2e6f821be14ab156c8d5428781084c16
I20260929 11:36:14.523034 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181: Bootstrap starting.
I20260929 11:36:14.527246 19962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:14.528703 19962 log.cc:824] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:14.532534 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181: No bootstrap required, opened a new log
Sep 29 11:36:14 dist-test-slave-c0l9 krb5kdc[19882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:14.542011 19962 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } }
I20260929 11:36:14.542562 19962 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:14.542704 19962 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fd6910a4ea40a89f64a08d720f4181, State: Initialized, Role: FOLLOWER
I20260929 11:36:14.543351 19962 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } }
I20260929 11:36:14.543628 19962 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:14.543767 19962 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:14.543941 19962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:14.544193 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:14.519688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.6.9.254:40045)
0929 11:36:14.520600 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:14.520613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:14.520683 (+ 70us) server_negotiation.cc:407] Connection header received
0929 11:36:14.522815 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:14.522842 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:14.523323 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:14.523698 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:14.525641 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:14.527160 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:14.529134 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:14.530281 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:14.530450 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:14.535762 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:14.535773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:14.535794 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:14.535836 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:14.539317 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:14.540327 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:14.540338 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:14.540345 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:14.540557 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:14.541021 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:14.541025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:14.541027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:14.541646 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:14.541894 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:14.542393 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 11:36:14.542909 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":432,"threads_started":1}
I20260929 11:36:14.546492 19962 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } }
I20260929 11:36:14.547129 19962 leader_election.cc:302] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fd6910a4ea40a89f64a08d720f4181; no voters:
I20260929 11:36:14.547806 19962 leader_election.cc:288] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:36:14.547910 19967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:14.548542 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:36:14.548527 19967 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [term 1 LEADER]: Becoming Leader. State: Replica: d2fd6910a4ea40a89f64a08d720f4181, State: Running, Role: LEADER
I20260929 11:36:14.549222 19967 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } }
I20260929 11:36:14.550614 19962 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:14.552562 19968 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fd6910a4ea40a89f64a08d720f4181. Latest consensus state: current_term: 1 leader_uuid: "d2fd6910a4ea40a89f64a08d720f4181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } } }
I20260929 11:36:14.553015 19968 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:14.553958 19972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:14.553777 19969 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fd6910a4ea40a89f64a08d720f4181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd6910a4ea40a89f64a08d720f4181" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40045 } } }
I20260929 11:36:14.554121 19969 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:14.559753 19972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:14.569610 19972 catalog_manager.cc:1347] Generated new cluster ID: 13e64114db3c4526baa2e56cafd83c4d
I20260929 11:36:14.569741 19972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:14.586179 19972 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:36:14.587558 19972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:14.599386 19972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fd6910a4ea40a89f64a08d720f4181: Generated new TSK 0
I20260929 11:36:14.600366 19972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:36:14.610772 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 19895
I20260929 11:36:14.612053 19957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:14.705312 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 19895
2026-09-29T11:36:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1917 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:36:14 dist-test-slave-c0l9 krb5kdc[19990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:36:14 dist-test-slave-c0l9 krb5kdc[19990](info): set up 2 sockets
Sep 29 11:36:14 dist-test-slave-c0l9 krb5kdc[19990](info): commencing operation
krb5kdc: starting...
W20260929 11:36:15.728886 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:36:15 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:36:15.963595 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:16.299427 20003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:16.299786 20003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:16.299916 20003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:16.307583 20003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:16.307729 20003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:16.307811 20003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:16.307893 20003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:16.307955 20003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:16.318220 20003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:16.319705 20003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:16.321122 20003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:16.334244 20008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:16.334141 20011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:16.334182 20009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:16.335404 20003 server_base.cc:1034] running on GCE node
I20260929 11:36:16.336284 20003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:16.337872 20003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:16.339123 20003 hybrid_clock.cc:648] HybridClock initialized: now 1790681776339066 us; error 48 us; skew 500 ppm
I20260929 11:36:16.339520 20003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:16 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681776, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:16.342947 20003 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:16.344568 20003 webserver.cc:460] Webserver started at http://127.6.9.254:44587/ using document root <none> and password file <none>
I20260929 11:36:16.345356 20003 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:16.345506 20003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:16.345813 20003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:16.349364 20003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "7a6a580c89b6434faf65b3e22397835b"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "8c77bad08c640ded85308e56a0230292"
server_key_iv: "fb8833bc641ff3a1eedff3f497353e82"
server_key_version: "encryptionkey@0"
I20260929 11:36:16.350296 20003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "7a6a580c89b6434faf65b3e22397835b"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "8c77bad08c640ded85308e56a0230292"
server_key_iv: "fb8833bc641ff3a1eedff3f497353e82"
server_key_version: "encryptionkey@0"
I20260929 11:36:16.356432 20003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:36:16.360225 20018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:16.361781 20003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:16.361972 20003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7a6a580c89b6434faf65b3e22397835b"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "8c77bad08c640ded85308e56a0230292"
server_key_iv: "fb8833bc641ff3a1eedff3f497353e82"
server_key_version: "encryptionkey@0"
I20260929 11:36:16.362169 20003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:16.394160 20003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:16.398849 20003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:16.399160 20003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:16.417366 20003 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40885
I20260929 11:36:16.417395 20069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40885 every 8 connection(s)
I20260929 11:36:16.418756 20003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:16.422540 20070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:16.427353 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20003
I20260929 11:36:16.427569 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:36:16.428378 6183 external_mini_cluster.cc:1428] Setting key a65d90faa64e27c7af1aa47c8a0928b8
I20260929 11:36:16.435024 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap starting.
I20260929 11:36:16.439018 20070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:16.440290 20070 log.cc:824] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:16.443624 20070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: No bootstrap required, opened a new log
Sep 29 11:36:16 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:16.452960 20070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:16.453398 20070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:16.453519 20070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Initialized, Role: FOLLOWER
I20260929 11:36:16.453948 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:16.430574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.6.9.254:40885)
0929 11:36:16.431542 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:16.431557 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:16.431599 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:36:16.433546 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:16.433573 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:16.433998 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:16.434336 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:16.436147 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:16.437700 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:16.439653 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:16.440399 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:16.440596 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:16.445037 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:16.445049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:16.445082 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:16.445161 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:16.448841 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:16.449696 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:16.449704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:16.449712 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:16.449926 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:16.450401 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:16.450406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:16.450409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:16.451432 (+ 1023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:16.451762 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:16.452290 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 11:36:16.452796 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":374,"threads_started":1}
I20260929 11:36:16.454386 20070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:16.454718 20070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:16.454859 20070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:16.455056 20070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:16.457831 20070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:16.458351 20070 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6a580c89b6434faf65b3e22397835b; no voters:
I20260929 11:36:16.458941 20070 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:36:16.459092 20075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:16.459726 20075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 LEADER]: Becoming Leader. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Running, Role: LEADER
I20260929 11:36:16.460232 20075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:16.461560 20070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:16.464095 20077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6a580c89b6434faf65b3e22397835b. Latest consensus state: current_term: 1 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:16.464514 20077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:16.464999 20076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:16.477790 20076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:16.480808 20082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:16.486008 20082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:16.495575 20082 catalog_manager.cc:1347] Generated new cluster ID: ceea1545804d4a768be97a9c4a9ce1f7
I20260929 11:36:16.495723 20082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:16.506531 20082 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:36:16.508478 20082 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:16.525607 20082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Generated new TSK 0
I20260929 11:36:16.526811 20082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:36:16.592036 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:16.937137 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:16.937443 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:16.937569 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:16.945134 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:16.945272 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:16.945416 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:16.955965 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:16.957530 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:16.958925 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:16.972810 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:16.972734 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:16.972767 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:16.978437 20096 server_base.cc:1034] running on GCE node
I20260929 11:36:16.979133 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:16.980171 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:16.981464 20096 hybrid_clock.cc:648] HybridClock initialized: now 1790681776981391 us; error 53 us; skew 500 ppm
I20260929 11:36:16.981810 20096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:16 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681776, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:16.985037 20096 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:16.986726 20096 webserver.cc:460] Webserver started at http://127.6.9.193:44105/ using document root <none> and password file <none>
I20260929 11:36:16.987461 20096 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:16.987581 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:16.987869 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:16.991007 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "e3872a64de51485ea7dc5435e233efec"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "d5ab63b913258d763a1ab2b5e98f1bbb"
server_key_iv: "dd2f4a0ce4cc6b204d67b8f54d0be1b3"
server_key_version: "encryptionkey@0"
I20260929 11:36:16.991811 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "e3872a64de51485ea7dc5435e233efec"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "d5ab63b913258d763a1ab2b5e98f1bbb"
server_key_iv: "dd2f4a0ce4cc6b204d67b8f54d0be1b3"
server_key_version: "encryptionkey@0"
I20260929 11:36:16.997085 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:36:17.000741 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.002342 20096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:36:17.002513 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "e3872a64de51485ea7dc5435e233efec"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "d5ab63b913258d763a1ab2b5e98f1bbb"
server_key_iv: "dd2f4a0ce4cc6b204d67b8f54d0be1b3"
server_key_version: "encryptionkey@0"
I20260929 11:36:17.002718 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:17.018769 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:17.023406 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:17.023717 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:17.024842 20096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:17.026288 20096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:17.026367 20096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.026496 20096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:17.026556 20096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.057201 20096 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:41087
I20260929 11:36:17.057257 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:41087 every 8 connection(s)
I20260929 11:36:17.058434 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:17.068111 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20096
I20260929 11:36:17.068333 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:36:17.069068 6183 external_mini_cluster.cc:1428] Setting key ff814993390fa75c1030989fc3a53191
Sep 29 11:36:17 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681776, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:17.078855 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:17.060801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:51289 (local address 127.6.9.254:40885)
0929 11:36:17.061243 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:17.061250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:17.062277 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 11:36:17.064040 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:17.064050 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:17.064129 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:17.064221 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:17.065950 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.066727 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.068526 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.068943 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.069034 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:17.072816 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:17.072821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:17.072824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:17.072858 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:17.075469 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:17.076320 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:17.076324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:17.076326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:17.076440 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:17.076919 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:17.076922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:17.076925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:17.077183 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:17.077293 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:17.078224 (+ 931us) server_negotiation.cc:299] Negotiation successful
0929 11:36:17.078492 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":192,"threads_started":1}
I20260929 11:36:17.079589 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:17.061367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:51289)
0929 11:36:17.062128 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:17.062171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:17.063799 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:17.064402 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:17.064409 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:17.065045 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:17.065794 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:17.065814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.066858 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.066862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:17.068394 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:17.068410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.069159 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.069167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:17.069451 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:17.070128 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:17.070150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:17.072523 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:17.075616 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:17.075625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:17.075644 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:17.076217 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:17.076529 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:17.076534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:17.076537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:17.076823 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:17.077292 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:17.077300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:17.077736 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 11:36:17.078058 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 11:36:17.078476 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260929 11:36:17.081169 20224 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:17.081511 20224 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:17.082298 20224 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:17.084519 20035 ts_manager.cc:194] Registered new tserver with Master: e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087)
I20260929 11:36:17.086071 20035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:51289
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:36:17.119840 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:17.455044 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:17.455399 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:17.455561 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:17.463481 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:17.463632 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:17.463786 20230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:17.474257 20230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:17.475625 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:17.477011 20230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:17.490231 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:17.490175 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:17.490206 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:17.491009 20230 server_base.cc:1034] running on GCE node
I20260929 11:36:17.491684 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:17.492750 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:17.493976 20230 hybrid_clock.cc:648] HybridClock initialized: now 1790681777493905 us; error 49 us; skew 500 ppm
I20260929 11:36:17.494303 20230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:17 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:17.498139 20230 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:17.500123 20230 webserver.cc:460] Webserver started at http://127.6.9.194:44477/ using document root <none> and password file <none>
I20260929 11:36:17.501137 20230 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:17.501286 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:17.501657 20230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:17.505395 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "475934661d1c00a9090196fb45b3707f"
server_key_iv: "46358439c2884f6a58d701fb0ad9a938"
server_key_version: "encryptionkey@0"
I20260929 11:36:17.506120 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "475934661d1c00a9090196fb45b3707f"
server_key_iv: "46358439c2884f6a58d701fb0ad9a938"
server_key_version: "encryptionkey@0"
I20260929 11:36:17.511338 20230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 11:36:17.514753 20245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.516096 20230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:36:17.516249 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "475934661d1c00a9090196fb45b3707f"
server_key_iv: "46358439c2884f6a58d701fb0ad9a938"
server_key_version: "encryptionkey@0"
I20260929 11:36:17.516430 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:17.532960 20230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:17.537259 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:17.537539 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:17.538581 20230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:17.540011 20230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:17.540092 20230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.540215 20230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:17.540266 20230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:17.570899 20230 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:40547
I20260929 11:36:17.570921 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:40547 every 8 connection(s)
I20260929 11:36:17.572214 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:17.582178 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20230
I20260929 11:36:17.582353 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:36:17.582932 6183 external_mini_cluster.cc:1428] Setting key 6d731e4c37362a83232bbcd16f995a55
Sep 29 11:36:17 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:17.593750 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:17.575007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56313 (local address 127.6.9.254:40885)
0929 11:36:17.575233 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:17.575239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:17.576395 (+ 1156us) server_negotiation.cc:407] Connection header received
0929 11:36:17.578242 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:17.578247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:17.578321 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:17.578413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:17.579989 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.580861 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.582679 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.583312 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.583445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:17.587325 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:17.587331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:17.587336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:17.587375 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:17.590485 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:17.591308 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:17.591314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:17.591317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:17.591471 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:17.591901 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:17.591906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:17.591909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:17.592258 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:17.592399 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:17.593145 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 11:36:17.593390 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:36:17.594588 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:17.575427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:56313)
0929 11:36:17.576227 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:17.576266 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:17.577946 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:17.578670 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:17.578679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:17.579215 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:17.579792 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:17.579810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.581024 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.581028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:17.582522 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:17.582536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:17.583527 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:17.583534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:17.583824 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:17.584493 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:17.584523 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:17.587081 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:17.590666 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:17.590675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:17.590690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:17.591191 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:17.591590 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:17.591594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:17.591596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:17.591804 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:17.592364 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:17.592370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:17.592766 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 11:36:17.593023 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 11:36:17.593451 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":318,"threads_started":1}
I20260929 11:36:17.596016 20358 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:17.596349 20358 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:17.597227 20358 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:17.598904 20035 ts_manager.cc:194] Registered new tserver with Master: a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547)
I20260929 11:36:17.599774 20035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:56313
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:36:17.635625 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:17.959174 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:17.959529 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:17.959685 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:17.967548 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:17.967684 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:17.967801 20363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:17.978380 20363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:17.979678 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:17.981063 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:17.994251 20368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:17.994204 20371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:17.994172 20369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:17.995380 20363 server_base.cc:1034] running on GCE node
I20260929 11:36:17.995987 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:17.996980 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:17.998234 20363 hybrid_clock.cc:648] HybridClock initialized: now 1790681777998182 us; error 47 us; skew 500 ppm
I20260929 11:36:17.998526 20363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:17 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:18.001667 20363 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:18.003132 20363 webserver.cc:460] Webserver started at http://127.6.9.195:35811/ using document root <none> and password file <none>
I20260929 11:36:18.003820 20363 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:18.003940 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:18.004241 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:18.007331 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "b7592e712f42470594fef5212e5e7d60"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "15736c85c1193725d75403d543b5f412"
server_key_iv: "90afc33a92a36f17a8067b917cac82b1"
server_key_version: "encryptionkey@0"
I20260929 11:36:18.008126 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "b7592e712f42470594fef5212e5e7d60"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "15736c85c1193725d75403d543b5f412"
server_key_iv: "90afc33a92a36f17a8067b917cac82b1"
server_key_version: "encryptionkey@0"
I20260929 11:36:18.013505 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 11:36:18.017189 20378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:18.018625 20363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:36:18.018792 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b7592e712f42470594fef5212e5e7d60"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "15736c85c1193725d75403d543b5f412"
server_key_iv: "90afc33a92a36f17a8067b917cac82b1"
server_key_version: "encryptionkey@0"
I20260929 11:36:18.018993 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:18.043891 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:18.048180 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:18.048465 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:18.049471 20363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:18.050869 20363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:18.050952 20363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:18.051088 20363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:18.051144 20363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:18.081884 20363 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:42991
I20260929 11:36:18.081923 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:42991 every 8 connection(s)
I20260929 11:36:18.083199 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:18.089010 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20363
I20260929 11:36:18.089207 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:36:18.089224 20224 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
I20260929 11:36:18.089789 6183 external_mini_cluster.cc:1428] Setting key 3f5946afeb331d0ffd7e29ff699fde38
Sep 29 11:36:18 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681777, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:18.103765 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.085773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:36095 (local address 127.6.9.254:40885)
0929 11:36:18.086001 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.086006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.087169 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 11:36:18.089091 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.089097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.089197 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.089309 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:18.091448 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.092484 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.094210 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.094689 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.094806 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.098181 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:18.098187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:18.098190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:18.098225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:18.100705 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.101442 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.101454 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.101456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.101596 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.101978 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.101981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.101984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.102276 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:18.102414 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.103150 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.103385 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:36:18.104828 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.086265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:36095)
0929 11:36:18.087001 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.087039 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.088772 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.089756 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.089770 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.090470 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:18.091252 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.091284 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.092629 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.092634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.094037 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.094051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.094875 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.094880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.095103 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.095755 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:18.095779 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:18.097970 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:18.100837 (+ 2867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:18.100844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:18.100858 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:18.101298 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:18.101679 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:18.101682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:18.101684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:18.101858 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:18.102408 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:18.102415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:18.102758 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.102992 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.103533 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":298,"threads_started":1}
I20260929 11:36:18.106086 20491 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:18.106369 20491 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:18.107080 20491 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:18.108507 20035 ts_manager.cc:194] Registered new tserver with Master: b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
I20260929 11:36:18.109364 20035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:36095
I20260929 11:36:18.118233 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:36:18.135113 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.122788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.6.9.254:40885)
0929 11:36:18.122956 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.122961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.123360 (+ 399us) server_negotiation.cc:407] Connection header received
0929 11:36:18.123485 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.123489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.123586 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.123711 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:18.125245 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.125962 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.127353 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.127784 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.127888 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.130283 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:18.130288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:18.130291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:18.130330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:18.132783 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.133413 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.133416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.133419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.133544 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.133843 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.133846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.133848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.134177 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:18.134353 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.134665 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.134837 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:36:18.139215 20035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:36:18.146102 20035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:36:18.169838 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.159141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.6.9.194:40547)
0929 11:36:18.160827 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.160838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.160870 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:36:18.160984 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.160994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.161275 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.161504 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:18.162639 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.163914 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.166948 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.168457 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.168589 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.168806 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.169384 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.169549 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1475,"thread_start_us":1355,"threads_started":1}
I20260929 11:36:18.169909 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:44082)
0929 11:36:18.161605 (+ 2174us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.161619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.161734 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.162368 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.162373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.162397 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:18.162843 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.162856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.164733 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.164738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.167038 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.167050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.168896 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.168903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.169093 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.169148 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.169254 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.169440 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"mutex_wait_us":17,"thread_start_us":1895,"threads_started":1}
I20260929 11:36:18.170643 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.159942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.6.9.193:41087)
0929 11:36:18.161167 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.161176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.161846 (+ 670us) server_negotiation.cc:407] Connection header received
0929 11:36:18.161942 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.161950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.162182 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.162396 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:18.163049 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.164556 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.167178 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.168116 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.168237 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.170192 (+ 1955us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.170265 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.170395 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":902,"threads_started":1}
I20260929 11:36:18.170022 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.158815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40547 (local address 127.0.0.1:44638)
0929 11:36:18.160117 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.160136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.160256 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.161814 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.161824 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.161861 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:18.162361 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.162375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.164060 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.164069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.166775 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.166791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.168982 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.168990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.169177 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.169214 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.169312 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.169456 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":538,"thread_start_us":937,"threads_started":1}
I20260929 11:36:18.170691 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.158151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42991 (local address 127.0.0.1:48106)
0929 11:36:18.159150 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.159195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.159428 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.163687 (+ 4259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.163692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.163720 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:18.164161 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.164177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.166068 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.166073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.168211 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.168227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.170127 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.170134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.170257 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.170281 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.170338 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.170433 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":35,"thread_start_us":496,"threads_started":1}
I20260929 11:36:18.171589 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.158482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.6.9.195:42991)
0929 11:36:18.163151 (+ 4669us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.163159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.163194 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:36:18.163283 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.163290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.163490 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.163677 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:18.164402 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.165923 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.168434 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.169886 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.170827 (+ 941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.171069 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.171142 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.171316 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4501,"thread_start_us":4336,"threads_started":1}
I20260929 11:36:18.174642 20293 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab563f825a7426eb9522cb028297edf (DEFAULT_TABLE table=test-table [id=29b396a303864508b6b505f74f632cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:18.174683 20159 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab563f825a7426eb9522cb028297edf (DEFAULT_TABLE table=test-table [id=29b396a303864508b6b505f74f632cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:18.175025 20426 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab563f825a7426eb9522cb028297edf (DEFAULT_TABLE table=test-table [id=29b396a303864508b6b505f74f632cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:18.175858 20293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab563f825a7426eb9522cb028297edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:18.176070 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab563f825a7426eb9522cb028297edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:18.176186 20426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab563f825a7426eb9522cb028297edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:18.187839 20509 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Bootstrap starting.
I20260929 11:36:18.190944 20511 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Bootstrap starting.
I20260929 11:36:18.193383 20509 tablet_bootstrap.cc:654] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:18.194661 20511 tablet_bootstrap.cc:654] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:18.194777 20509 log.cc:824] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:18.195451 20510 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Bootstrap starting.
I20260929 11:36:18.196012 20511 log.cc:824] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:18.198793 20510 tablet_bootstrap.cc:654] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:18.199729 20510 log.cc:824] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:18.202486 20510 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: No bootstrap required, opened a new log
I20260929 11:36:18.202769 20510 ts_tablet_manager.cc:1397] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:36:18.202790 20509 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: No bootstrap required, opened a new log
I20260929 11:36:18.202803 20511 tablet_bootstrap.cc:492] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: No bootstrap required, opened a new log
I20260929 11:36:18.203052 20511 ts_tablet_manager.cc:1397] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 11:36:18.203052 20509 ts_tablet_manager.cc:1397] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260929 11:36:18.210614 20510 raft_consensus.cc:348] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.210988 20510 raft_consensus.cc:374] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:18.211100 20510 raft_consensus.cc:729] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3872a64de51485ea7dc5435e233efec, State: Initialized, Role: FOLLOWER
I20260929 11:36:18.211720 20510 consensus_queue.cc:260] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.213557 20510 ts_tablet_manager.cc:1428] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260929 11:36:18.213376 20511 raft_consensus.cc:348] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.213376 20509 raft_consensus.cc:348] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.213832 20509 raft_consensus.cc:374] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:18.213853 20511 raft_consensus.cc:374] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:18.213938 20511 raft_consensus.cc:729] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7592e712f42470594fef5212e5e7d60, State: Initialized, Role: FOLLOWER
I20260929 11:36:18.213956 20509 raft_consensus.cc:729] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3656c32e4f049a5b91bd71c73d2c27f, State: Initialized, Role: FOLLOWER
I20260929 11:36:18.214561 20511 consensus_queue.cc:260] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.214560 20509 consensus_queue.cc:260] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.215710 20358 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
I20260929 11:36:18.216449 20509 ts_tablet_manager.cc:1428] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 11:36:18.218430 20511 ts_tablet_manager.cc:1428] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 11:36:18.218528 20491 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
W20260929 11:36:18.310739 20225 tablet.cc:2369] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:36:18.324385 20359 tablet.cc:2369] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:36:18.334839 20492 tablet.cc:2369] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:18.640645 20516 raft_consensus.cc:482] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:18.640969 20516 raft_consensus.cc:504] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.642508 20516 leader_election.cc:288] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087), a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547)
I20260929 11:36:18.649116 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.642973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:58081 (local address 127.6.9.194:40547)
0929 11:36:18.643223 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.643231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.643747 (+ 516us) server_negotiation.cc:407] Connection header received
0929 11:36:18.643877 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.643882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.643952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.644045 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:18.644794 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.645535 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.647379 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.648196 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.648319 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.648420 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.648723 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.648865 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:36:18.649121 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.643017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:35591 (local address 127.6.9.193:41087)
0929 11:36:18.643220 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.643226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.643607 (+ 381us) server_negotiation.cc:407] Connection header received
0929 11:36:18.643802 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.643807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.643894 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.644005 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:18.644849 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.645575 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.647382 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.648191 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.648318 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.648424 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.648750 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.648881 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:36:18.649247 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.642842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40547 (local address 127.6.9.195:58081)
0929 11:36:18.643652 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.643667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.643770 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.644074 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.644079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.644111 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:18.644600 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.644612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.645677 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.645683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.647209 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.647224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.648367 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.648373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.648536 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.648563 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.648641 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.648803 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":25,"thread_start_us":510,"threads_started":1}
I20260929 11:36:18.649307 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.642825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.6.9.195:35591)
0929 11:36:18.643476 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:18.643500 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:18.643661 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:18.644081 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:18.644085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:18.644123 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:18.644598 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.644612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.645721 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.645726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.647211 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:18.647225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.648420 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.648426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:18.648580 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.648604 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:36:18.648722 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 11:36:18.648857 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":8,"spinlock_wait_cycles":57472,"thread_start_us":414,"threads_started":1}
I20260929 11:36:18.650480 20313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab563f825a7426eb9522cb028297edf" candidate_uuid: "b7592e712f42470594fef5212e5e7d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" is_pre_election: true
I20260929 11:36:18.650424 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab563f825a7426eb9522cb028297edf" candidate_uuid: "b7592e712f42470594fef5212e5e7d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3872a64de51485ea7dc5435e233efec" is_pre_election: true
I20260929 11:36:18.651042 20179 raft_consensus.cc:2454] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7592e712f42470594fef5212e5e7d60 in term 0.
I20260929 11:36:18.651054 20313 raft_consensus.cc:2454] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7592e712f42470594fef5212e5e7d60 in term 0.
I20260929 11:36:18.651751 20382 leader_election.cc:302] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7592e712f42470594fef5212e5e7d60, e3872a64de51485ea7dc5435e233efec; no voters:
I20260929 11:36:18.652143 20516 raft_consensus.cc:2781] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:36:18.652278 20516 raft_consensus.cc:482] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:36:18.652377 20516 raft_consensus.cc:3037] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:18.654625 20516 raft_consensus.cc:504] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.655225 20516 leader_election.cc:288] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [CANDIDATE]: Term 1 election: Requested vote from peers e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087), a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547)
I20260929 11:36:18.655823 20313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab563f825a7426eb9522cb028297edf" candidate_uuid: "b7592e712f42470594fef5212e5e7d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
I20260929 11:36:18.655881 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab563f825a7426eb9522cb028297edf" candidate_uuid: "b7592e712f42470594fef5212e5e7d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3872a64de51485ea7dc5435e233efec"
I20260929 11:36:18.656060 20313 raft_consensus.cc:3037] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:18.656080 20179 raft_consensus.cc:3037] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:18.658320 20179 raft_consensus.cc:2454] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7592e712f42470594fef5212e5e7d60 in term 1.
I20260929 11:36:18.658327 20313 raft_consensus.cc:2454] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7592e712f42470594fef5212e5e7d60 in term 1.
I20260929 11:36:18.658790 20382 leader_election.cc:302] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7592e712f42470594fef5212e5e7d60, e3872a64de51485ea7dc5435e233efec; no voters:
I20260929 11:36:18.659122 20516 raft_consensus.cc:2781] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:18.659551 20516 raft_consensus.cc:686] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 1 LEADER]: Becoming Leader. State: Replica: b7592e712f42470594fef5212e5e7d60, State: Running, Role: LEADER
I20260929 11:36:18.660313 20516 consensus_queue.cc:237] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } }
I20260929 11:36:18.665408 20035 catalog_manager.cc:5495] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7592e712f42470594fef5212e5e7d60 (127.6.9.195). New cstate: current_term: 1 leader_uuid: "b7592e712f42470594fef5212e5e7d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } health_report { overall_health: UNKNOWN } } }
I20260929 11:36:18.745687 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.739884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.6.9.195:42991)
0929 11:36:18.740575 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.740581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.740604 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:36:18.740734 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.740738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.740826 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.740969 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:36:18.741408 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.742092 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.743501 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.743894 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.744011 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.744496 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:36:18.745073 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:36:18.745216 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.745308 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.745398 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":442,"threads_started":1}
I20260929 11:36:18.753649 20179 raft_consensus.cc:1261] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Refusing update from remote peer b7592e712f42470594fef5212e5e7d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:18.753649 20313 raft_consensus.cc:1261] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Refusing update from remote peer b7592e712f42470594fef5212e5e7d60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:18.754626 20520 consensus_queue.cc:1035] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:18.754995 20516 consensus_queue.cc:1035] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:18.766059 20528 mvcc.cc:205] Tried to move back new op lower bound from 7334632565765197824 to 7334632565397245952. Current Snapshot: MvccSnapshot[applied={T|T < 7334632565765197824}]
I20260929 11:36:18.766729 20532 mvcc.cc:205] Tried to move back new op lower bound from 7334632565765197824 to 7334632565397245952. Current Snapshot: MvccSnapshot[applied={T|T < 7334632565765197824}]
I20260929 11:36:18.779316 20034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:18.779598 20034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:18.783351 20034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Sending DeleteTablet for 3 replicas of tablet 5ab563f825a7426eb9522cb028297edf
I20260929 11:36:18.784596 20293 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab563f825a7426eb9522cb028297edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:18 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:44638
I20260929 11:36:18.784602 20426 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab563f825a7426eb9522cb028297edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:18 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:48106
I20260929 11:36:18.785179 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20096
I20260929 11:36:18.785435 20159 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab563f825a7426eb9522cb028297edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:18 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:44082
I20260929 11:36:18.796041 20542 tablet_replica.cc:330] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: stopping tablet replica
I20260929 11:36:18.797381 20542 raft_consensus.cc:2229] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:18.798727 20542 raft_consensus.cc:2258] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:18.801306 20540 tablet_replica.cc:330] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: stopping tablet replica
I20260929 11:36:18.802032 20540 raft_consensus.cc:2229] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:18.802659 20542 ts_tablet_manager.cc:1905] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:18.802724 20540 raft_consensus.cc:2258] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:18.803561 20541 tablet_replica.cc:330] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: stopping tablet replica
I20260929 11:36:18.804103 20541 raft_consensus.cc:2229] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:36:18.804881 20540 ts_tablet_manager.cc:1905] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:18.804855 20541 raft_consensus.cc:2258] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:18.806599 20541 ts_tablet_manager.cc:1905] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:18.812426 20540 ts_tablet_manager.cc:1918] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:18.812737 20540 log.cc:1196] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/5ab563f825a7426eb9522cb028297edf
I20260929 11:36:18.813352 20540 ts_tablet_manager.cc:1939] T 5ab563f825a7426eb9522cb028297edf P a3656c32e4f049a5b91bd71c73d2c27f: Deleting consensus metadata
I20260929 11:36:18.814211 20541 ts_tablet_manager.cc:1918] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:18.814432 20541 log.cc:1196] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/5ab563f825a7426eb9522cb028297edf
I20260929 11:36:18.815032 20541 ts_tablet_manager.cc:1939] T 5ab563f825a7426eb9522cb028297edf P b7592e712f42470594fef5212e5e7d60: Deleting consensus metadata
I20260929 11:36:18.815613 20020 catalog_manager.cc:4841] TS a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): tablet 5ab563f825a7426eb9522cb028297edf (table test-table [id=29b396a303864508b6b505f74f632cbc]) successfully deleted
I20260929 11:36:18.816834 20022 catalog_manager.cc:4841] TS b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): tablet 5ab563f825a7426eb9522cb028297edf (table test-table [id=29b396a303864508b6b505f74f632cbc]) successfully deleted
Sep 29 11:36:18 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:36:18.819634 20542 ts_tablet_manager.cc:1918] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:18.819808 20542 log.cc:1196] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/5ab563f825a7426eb9522cb028297edf
I20260929 11:36:18.820338 20542 ts_tablet_manager.cc:1939] T 5ab563f825a7426eb9522cb028297edf P e3872a64de51485ea7dc5435e233efec: Deleting consensus metadata
I20260929 11:36:18.822108 20022 catalog_manager.cc:4841] TS e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087): tablet 5ab563f825a7426eb9522cb028297edf (table test-table [id=29b396a303864508b6b505f74f632cbc]) successfully deleted
I20260929 11:36:18.826982 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.786479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.6.9.193:41087)
0929 11:36:18.786652 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.786658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.799697 (+ 13039us) server_negotiation.cc:407] Connection header received
0929 11:36:18.800631 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.800639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.800777 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.800866 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:18.806324 (+ 5458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.807461 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.812624 (+ 5163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.813375 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.813507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.819512 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:18.819534 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:18.819556 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:18.819610 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:18.823772 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.824570 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.824581 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.824584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.824812 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.825178 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.825183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.825186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.825693 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:18.825996 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.826361 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.826598 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:36:18.827726 20219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:18.916798 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20096
I20260929 11:36:18.932626 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20230
Sep 29 11:36:18 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:36:18.949290 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:18.933503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.6.9.194:40547)
0929 11:36:18.933738 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:18.933744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:18.933768 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:36:18.933841 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:18.933846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:18.933946 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:18.934060 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:18.935253 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.936439 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.937846 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:18.938498 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:18.938622 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:18.943204 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:18.943221 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:18.943237 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:18.943280 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:18.946091 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.946989 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.946998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.947000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.947158 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:18.947587 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:18.947591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:18.947593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:18.948107 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:18.948378 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:18.948765 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 11:36:18.948973 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:36:18.950085 20353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:19.039598 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20230
I20260929 11:36:19.059587 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20363
Sep 29 11:36:19 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:36:19.076601 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:19.060390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.6.9.195:42991)
0929 11:36:19.060568 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:19.060574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:19.060691 (+ 117us) server_negotiation.cc:407] Connection header received
0929 11:36:19.060875 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:19.060879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:19.060945 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:19.061037 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:19.062639 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:19.063460 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:19.064874 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:19.065442 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:19.065568 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:19.070288 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:19.070303 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:19.070323 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:19.070375 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:19.073611 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:19.074393 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:19.074401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:19.074403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:19.074560 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:19.074974 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:19.074977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:19.074979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:19.075359 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:19.075680 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:19.075987 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 11:36:19.076244 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:36:19.077425 20486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:19.201915 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20363
I20260929 11:36:19.218010 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20003
I20260929 11:36:19.219202 20065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:19.329174 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20003
I20260929 11:36:19.346938 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--webserver_interface=127.6.9.254
--webserver_port=44587
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:19.686170 20548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:19.686496 20548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:19.686635 20548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:19.694427 20548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:19.694559 20548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:19.694634 20548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:19.694715 20548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:19.694772 20548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:19.705024 20548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=44587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:19.706409 20548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:19.707746 20548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:19.729995 20554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:19.729954 20557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:19.729909 20555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:19.730904 20548 server_base.cc:1034] running on GCE node
I20260929 11:36:19.731850 20548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:19.733647 20548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:19.734887 20548 hybrid_clock.cc:648] HybridClock initialized: now 1790681779734820 us; error 60 us; skew 500 ppm
I20260929 11:36:19.735309 20548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:19 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681779, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:19.745345 20548 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:19.747293 20548 webserver.cc:460] Webserver started at http://127.6.9.254:44587/ using document root <none> and password file <none>
I20260929 11:36:19.748145 20548 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:19.748324 20548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:19.754912 20548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:36:19.758265 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:19.759783 20548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:36:19.759979 20548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7a6a580c89b6434faf65b3e22397835b"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "8c77bad08c640ded85308e56a0230292"
server_key_iv: "fb8833bc641ff3a1eedff3f497353e82"
server_key_version: "encryptionkey@0"
I20260929 11:36:19.761170 20548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:19.777464 20548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:19.781544 20548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:19.781816 20548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:19.800268 20548 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40885
I20260929 11:36:19.800288 20615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40885 every 8 connection(s)
I20260929 11:36:19.801910 20548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:19.809262 20616 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:36:19.809898 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20548
I20260929 11:36:19.810663 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:41087
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=44105
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:19.813264 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap starting.
I20260929 11:36:19.838940 20616 log.cc:824] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:19.855093 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:36:19.855751 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap complete.
I20260929 11:36:19.869422 20616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:19.870353 20616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Initialized, Role: FOLLOWER
I20260929 11:36:19.871021 20616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:19.871258 20616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:19.871376 20616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:19.871536 20616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:36:19.875697 20616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:19.876281 20616 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6a580c89b6434faf65b3e22397835b; no voters:
I20260929 11:36:19.876924 20616 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:36:19.877110 20620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:36:19.878970 20616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:19.879053 20620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 LEADER]: Becoming Leader. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Running, Role: LEADER
I20260929 11:36:19.879668 20620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:19.882843 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6a580c89b6434faf65b3e22397835b. Latest consensus state: current_term: 2 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:19.883834 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:19.882623 20621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:19.884292 20621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:19.896554 20629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:19.906998 20629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:19.908751 20629 catalog_manager.cc:1259] Loaded cluster ID: ceea1545804d4a768be97a9c4a9ce1f7
I20260929 11:36:19.908980 20629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:19.913201 20629 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 11:36:19.934120 20629 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 11:36:19.954289 20629 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:19.956230 20629 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Loaded TSK: 0
I20260929 11:36:19.957947 20629 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:36:20.198604 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:20.198915 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:20.199059 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:20.206730 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:20.206861 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:20.207012 20618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:20.217633 20618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:41087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=44105
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:20.219130 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:20.220533 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:20.236706 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:20.237021 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:20.237365 20618 server_base.cc:1034] running on GCE node
W20260929 11:36:20.237501 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:20.238391 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:20.239733 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:20.240988 20618 hybrid_clock.cc:648] HybridClock initialized: now 1790681780240930 us; error 43 us; skew 500 ppm
I20260929 11:36:20.241346 20618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:20 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:20.244750 20618 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:20.246443 20618 webserver.cc:460] Webserver started at http://127.6.9.193:44105/ using document root <none> and password file <none>
I20260929 11:36:20.247225 20618 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:20.247366 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:20.253526 20618 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:36:20.256738 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.258414 20618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:36:20.258598 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "e3872a64de51485ea7dc5435e233efec"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "d5ab63b913258d763a1ab2b5e98f1bbb"
server_key_iv: "dd2f4a0ce4cc6b204d67b8f54d0be1b3"
server_key_version: "encryptionkey@0"
I20260929 11:36:20.259657 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:20.276311 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:20.280839 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:20.281106 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:20.282083 20618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:20.283420 20618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:20.283516 20618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.283625 20618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:20.283701 20618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.312474 20618 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:41087
I20260929 11:36:20.312541 20765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:41087 every 8 connection(s)
I20260929 11:36:20.314217 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:20.318578 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20618
I20260929 11:36:20.319228 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:40547
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=44477
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:20 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:20.339967 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:20.317554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:50885)
0929 11:36:20.318372 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:20.318419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:20.320272 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:20.322195 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:20.322206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:20.322880 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:20.323570 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:20.323593 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.325566 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.325571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:20.327140 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:20.327174 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.328339 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.328346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:20.328624 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:20.329372 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:20.329396 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:20.331830 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:20.335424 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:20.335434 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:20.335452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:20.335927 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:20.336418 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:20.336423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:20.336427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:20.336638 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:20.337753 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:20.337760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:20.338132 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 11:36:20.338379 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 11:36:20.338839 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":365,"threads_started":1}
I20260929 11:36:20.340330 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:20.317423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:50885 (local address 127.6.9.254:40885)
0929 11:36:20.318017 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:20.318030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:20.318573 (+ 543us) server_negotiation.cc:407] Connection header received
0929 11:36:20.320916 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:20.320968 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:20.321448 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:20.321967 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:20.323845 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.325381 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.327335 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.328066 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.328277 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:20.332161 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:20.332171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:20.332189 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:20.332232 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:20.335235 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:20.336087 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:20.336095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:20.336101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:20.336286 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:20.336811 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:20.336816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:20.336820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:20.337576 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:20.337793 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:20.338646 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 11:36:20.339119 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260929 11:36:20.343470 20766 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:20.343806 20766 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:20.344703 20766 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:20.347486 20581 ts_manager.cc:194] Registered new tserver with Master: e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087)
I20260929 11:36:20.349635 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:50885
W20260929 11:36:20.654101 20770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:20.654383 20770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:20.654492 20770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:20.661741 20770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:20.661856 20770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:20.661968 20770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:20.671993 20770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:40547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=44477
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:20.673334 20770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:20.674628 20770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:20.687327 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:20.687326 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:20.687362 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:20.688367 20770 server_base.cc:1034] running on GCE node
I20260929 11:36:20.688990 20770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:20.689936 20770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:20.691136 20770 hybrid_clock.cc:648] HybridClock initialized: now 1790681780691090 us; error 38 us; skew 500 ppm
I20260929 11:36:20.691435 20770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:20 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:20.694445 20770 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:20.695871 20770 webserver.cc:460] Webserver started at http://127.6.9.194:44477/ using document root <none> and password file <none>
I20260929 11:36:20.696503 20770 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:20.696624 20770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:20.701786 20770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 11:36:20.704368 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.705684 20770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:36:20.705821 20770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "475934661d1c00a9090196fb45b3707f"
server_key_iv: "46358439c2884f6a58d701fb0ad9a938"
server_key_version: "encryptionkey@0"
I20260929 11:36:20.706733 20770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:20.726234 20770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:20.730639 20770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:20.730901 20770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:20.731894 20770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:20.733373 20770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:20.733461 20770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.733549 20770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:20.733620 20770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:20.764007 20770 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:40547
I20260929 11:36:20.764091 20897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:40547 every 8 connection(s)
I20260929 11:36:20.765592 20770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:20.773780 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20770
I20260929 11:36:20.774371 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:42991
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=35811
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:36:20 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:20.788079 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:20.768184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:38147 (local address 127.6.9.254:40885)
0929 11:36:20.768398 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:20.768404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:20.769716 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 11:36:20.771600 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:20.771606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:20.771683 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:20.771781 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:20.773513 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.774301 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.776638 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.777143 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.777255 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:20.781665 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:20.781671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:20.781675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:20.781713 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:20.784761 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:20.785594 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:20.785598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:20.785601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:20.785750 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:20.786222 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:20.786226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:20.786228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:20.786536 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:20.786679 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:20.787459 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 11:36:20.787725 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:36:20.789227 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:20.768765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:38147)
0929 11:36:20.769534 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:20.769576 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:20.771288 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:20.772024 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:20.772034 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:20.772599 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:20.773267 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:20.773289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.774489 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.774495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:20.776430 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:20.776447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:20.777388 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:20.777395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:20.777712 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:20.778468 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:20.778500 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:20.781398 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:20.784932 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:20.784940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:20.784956 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:20.785429 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:20.785870 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:20.785874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:20.785876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:20.786088 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:20.786679 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:20.786689 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:20.787064 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 11:36:20.787325 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 11:36:20.787914 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":310,"threads_started":1}
I20260929 11:36:20.790982 20898 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:20.791347 20898 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:20.792147 20898 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:20.793922 20581 ts_manager.cc:194] Registered new tserver with Master: a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547)
I20260929 11:36:20.795109 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:38147
W20260929 11:36:21.105706 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:21.106045 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:21.106151 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:21.113705 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:21.113837 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:21.113953 20901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:21.123950 20901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:42991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=35811
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:21.125306 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:21.126664 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:21.139647 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:21.139686 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:21.139623 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:21.140457 20901 server_base.cc:1034] running on GCE node
I20260929 11:36:21.141062 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:21.141919 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:21.143110 20901 hybrid_clock.cc:648] HybridClock initialized: now 1790681781143052 us; error 52 us; skew 500 ppm
I20260929 11:36:21.143412 20901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:21 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:21.146430 20901 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:21.147876 20901 webserver.cc:460] Webserver started at http://127.6.9.195:35811/ using document root <none> and password file <none>
I20260929 11:36:21.148524 20901 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:21.148633 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:21.154060 20901 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 11:36:21.156780 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:21.158156 20901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:21.158310 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b7592e712f42470594fef5212e5e7d60"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "15736c85c1193725d75403d543b5f412"
server_key_iv: "90afc33a92a36f17a8067b917cac82b1"
server_key_version: "encryptionkey@0"
I20260929 11:36:21.159273 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:21.182559 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:21.188238 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:21.188548 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:21.189855 20901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:21.191635 20901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:21.191761 20901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:21.191875 20901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:21.191951 20901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:21.226447 20901 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:42991
I20260929 11:36:21.226491 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:42991 every 8 connection(s)
I20260929 11:36:21.227900 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:21.229894 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 20901
Sep 29 11:36:21 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:21.246454 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.231147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.6.9.254:40885)
0929 11:36:21.231563 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.231568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.231591 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:36:21.231729 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.231734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.231817 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.231939 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:21.233626 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.234769 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.236136 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.236605 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.236734 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.239779 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:21.239789 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:21.239794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:21.239857 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:21.243193 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.244133 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.244139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.244142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.244302 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.244739 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.244744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.244748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.245118 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:21.245411 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.245768 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.245993 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":31232,"thread_start_us":299,"threads_started":1}
I20260929 11:36:21.251151 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.230672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:49437 (local address 127.6.9.254:40885)
0929 11:36:21.230881 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.230887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.232216 (+ 1329us) server_negotiation.cc:407] Connection header received
0929 11:36:21.234376 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.234380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.234444 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.234525 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:21.236180 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.237111 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.238875 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.239533 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.239665 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.244204 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:21.244210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:21.244215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:21.244251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:21.247353 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.248741 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.248744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.248747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.248867 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.249421 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.249426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.249430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.249731 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:21.249847 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.250674 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.250867 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"spinlock_wait_cycles":9344}
I20260929 11:36:21.252058 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.231157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:49437)
0929 11:36:21.232068 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.232124 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.234063 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.234825 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.234837 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.235393 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:21.235989 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.236007 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.237283 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.237287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.238723 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.238737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.239806 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.239815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.240163 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.240929 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:21.240964 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:21.243929 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:21.247868 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:21.247879 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:21.247899 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:21.248593 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:21.248987 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:21.248992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:21.248995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:21.249300 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:21.249840 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:21.249847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:21.250228 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.250479 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.250902 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":392,"threads_started":1}
I20260929 11:36:21.253682 21029 heartbeater.cc:347] Connected to a master server at 127.6.9.254:40885
I20260929 11:36:21.253986 21029 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:21.254736 21029 heartbeater.cc:511] Master 127.6.9.254:40885 requested a full tablet report, sending...
I20260929 11:36:21.256201 20581 ts_manager.cc:194] Registered new tserver with Master: b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
I20260929 11:36:21.257083 20581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:49437
I20260929 11:36:21.261879 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:36:21.269495 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.262799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.6.9.254:40885)
0929 11:36:21.262956 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.262960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.263438 (+ 478us) server_negotiation.cc:407] Connection header received
0929 11:36:21.263655 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.263661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.263828 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.263964 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:36:21.264459 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.265479 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.267077 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.267539 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.267648 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.268253 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:36:21.268894 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:36:21.269020 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.269193 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.269277 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:36:21.270632 20581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:36:21.277997 20581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:36:21.302875 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.290804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.6.9.194:40547)
0929 11:36:21.293297 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.293305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.293333 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:36:21.293436 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.293445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.293696 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.294732 (+ 1036us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:21.294970 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.296615 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.299971 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301498 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.302018 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.302220 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.302343 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.302516 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":2241,"threads_started":1}
I20260929 11:36:21.302908 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.290786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40547 (local address 127.0.0.1:47344)
0929 11:36:21.292721 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.292739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.292852 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.294039 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.294045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.294078 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:21.294510 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.294521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.296827 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.296833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.299783 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.299798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301753 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.301761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.302056 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.302155 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.302290 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.302511 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"mutex_wait_us":271,"thread_start_us":602,"threads_started":1}
I20260929 11:36:21.302920 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.291191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.6.9.193:41087)
0929 11:36:21.292819 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.292829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.292861 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:36:21.292968 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.292976 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.293269 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.294311 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:21.294556 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.296239 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.300179 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301564 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.301696 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.301915 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.302439 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.302608 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":1366,"threads_started":1}
I20260929 11:36:21.303826 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.290763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42860)
0929 11:36:21.292028 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.292097 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.292276 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.293521 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.293530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.293565 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:21.294045 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.294059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.296425 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.296431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.299960 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.299973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301808 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.301814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.301994 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.302152 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.302277 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.303485 (+ 1208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":1175,"thread_start_us":1630,"threads_started":1}
I20260929 11:36:21.307521 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.293374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42991 (local address 127.0.0.1:39130)
0929 11:36:21.298996 (+ 5622us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.299015 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.299141 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.299848 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.299853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.299880 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:21.300231 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.300239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301595 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.301600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.304936 (+ 3336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.304951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.306831 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.306837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.306961 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.306985 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.307071 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.307200 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5475,"thread_start_us":5317,"threads_started":1}
I20260929 11:36:21.307648 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.293168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.6.9.195:42991)
0929 11:36:21.298456 (+ 5288us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.298462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.299274 (+ 812us) server_negotiation.cc:407] Connection header received
0929 11:36:21.299385 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.299394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.299660 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.299862 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:21.300401 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.301467 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.305103 (+ 3636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.306313 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.306434 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.306672 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.307143 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.307306 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5126,"thread_start_us":4977,"threads_started":1}
I20260929 11:36:21.307461 20833 tablet_service.cc:1467] Processing CreateTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 (DEFAULT_TABLE table=test-table [id=79b80185be0e451fa7fbee93c66ae409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:21.307461 20701 tablet_service.cc:1467] Processing CreateTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 (DEFAULT_TABLE table=test-table [id=79b80185be0e451fa7fbee93c66ae409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:21.308965 20833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d3a42ed054d3c99abc0a3b0ad1068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:21.308955 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d3a42ed054d3c99abc0a3b0ad1068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:21.311067 20964 tablet_service.cc:1467] Processing CreateTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 (DEFAULT_TABLE table=test-table [id=79b80185be0e451fa7fbee93c66ae409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:21.312458 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370d3a42ed054d3c99abc0a3b0ad1068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:21.325268 21046 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Bootstrap starting.
I20260929 11:36:21.326133 21045 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Bootstrap starting.
I20260929 11:36:21.330066 21045 tablet_bootstrap.cc:654] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:21.331486 21045 log.cc:824] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:21.332197 21047 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Bootstrap starting.
I20260929 11:36:21.332242 21046 tablet_bootstrap.cc:654] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:21.333648 21046 log.cc:824] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:21.336414 21047 tablet_bootstrap.cc:654] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:21.337509 21046 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: No bootstrap required, opened a new log
I20260929 11:36:21.337785 21046 ts_tablet_manager.cc:1397] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260929 11:36:21.337836 21045 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: No bootstrap required, opened a new log
I20260929 11:36:21.337960 21047 log.cc:824] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:21.338143 21045 ts_tablet_manager.cc:1397] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260929 11:36:21.341460 21047 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: No bootstrap required, opened a new log
I20260929 11:36:21.341740 21047 ts_tablet_manager.cc:1397] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 11:36:21.345772 21046 raft_consensus.cc:348] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.346516 21046 raft_consensus.cc:374] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:21.346653 21046 raft_consensus.cc:729] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3656c32e4f049a5b91bd71c73d2c27f, State: Initialized, Role: FOLLOWER
I20260929 11:36:21.347301 21046 consensus_queue.cc:260] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.348506 20898 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
I20260929 11:36:21.348798 21045 raft_consensus.cc:348] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.349253 21045 raft_consensus.cc:374] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:21.349308 21046 ts_tablet_manager.cc:1428] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260929 11:36:21.349339 21045 raft_consensus.cc:729] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3872a64de51485ea7dc5435e233efec, State: Initialized, Role: FOLLOWER
I20260929 11:36:21.349979 21045 consensus_queue.cc:260] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.351186 20766 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
I20260929 11:36:21.351882 21045 ts_tablet_manager.cc:1428] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 11:36:21.353379 21047 raft_consensus.cc:348] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.353819 21047 raft_consensus.cc:374] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:21.353935 21047 raft_consensus.cc:729] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7592e712f42470594fef5212e5e7d60, State: Initialized, Role: FOLLOWER
I20260929 11:36:21.354611 21047 consensus_queue.cc:260] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.355775 21029 heartbeater.cc:503] Master 127.6.9.254:40885 was elected leader, sending a full tablet report...
I20260929 11:36:21.356357 21047 ts_tablet_manager.cc:1428] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 11:36:21.358263 21051 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:21.358517 21051 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.360235 21051 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
I20260929 11:36:21.366876 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.360606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:45555 (local address 127.6.9.193:41087)
0929 11:36:21.360806 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.360810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.361150 (+ 340us) server_negotiation.cc:407] Connection header received
0929 11:36:21.361339 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.361344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.361424 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.361529 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:21.362026 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.362825 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.365012 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.365880 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.366017 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.366128 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.366466 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.366612 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:36:21.366942 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.360408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.6.9.194:45555)
0929 11:36:21.361028 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.361046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.361188 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.361557 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.361563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.361590 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:21.361887 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.361897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.362954 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.362958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.364828 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.364846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.366136 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.366144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.366304 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.366330 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.366421 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.366568 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":304,"threads_started":1}
I20260929 11:36:21.372774 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.361135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42991 (local address 127.6.9.194:34661)
0929 11:36:21.367173 (+ 6038us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.367189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:21.367327 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:21.367642 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:21.367646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:21.367674 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:21.367843 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.367851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.368862 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.368867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.370238 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:21.370290 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.372099 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.372103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:21.372263 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.372290 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:36:21.372372 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 11:36:21.372491 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5943,"threads_started":1}
I20260929 11:36:21.372889 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.361345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34661 (local address 127.6.9.195:42991)
0929 11:36:21.361572 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.361579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.367279 (+ 5700us) server_negotiation.cc:407] Connection header received
0929 11:36:21.367460 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.367464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.367535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.367609 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:21.367975 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.368740 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.370488 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.371890 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.372045 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.372167 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.372419 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.372577 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:36:21.373997 20984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d3a42ed054d3c99abc0a3b0ad1068" candidate_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7592e712f42470594fef5212e5e7d60" is_pre_election: true
I20260929 11:36:21.374475 20721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d3a42ed054d3c99abc0a3b0ad1068" candidate_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3872a64de51485ea7dc5435e233efec" is_pre_election: true
I20260929 11:36:21.374675 20984 raft_consensus.cc:2454] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3656c32e4f049a5b91bd71c73d2c27f in term 0.
I20260929 11:36:21.375021 20721 raft_consensus.cc:2454] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3656c32e4f049a5b91bd71c73d2c27f in term 0.
I20260929 11:36:21.375509 20789 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60; no voters:
I20260929 11:36:21.375911 21051 raft_consensus.cc:2781] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:36:21.376070 21051 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:36:21.376178 21051 raft_consensus.cc:3037] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:21.378662 21051 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.379324 21051 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [CANDIDATE]: Term 1 election: Requested vote from peers e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
I20260929 11:36:21.379990 20721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d3a42ed054d3c99abc0a3b0ad1068" candidate_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3872a64de51485ea7dc5435e233efec"
I20260929 11:36:21.380005 20984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370d3a42ed054d3c99abc0a3b0ad1068" candidate_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7592e712f42470594fef5212e5e7d60"
I20260929 11:36:21.380208 20984 raft_consensus.cc:3037] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:21.380209 20721 raft_consensus.cc:3037] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:21.382540 20984 raft_consensus.cc:2454] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3656c32e4f049a5b91bd71c73d2c27f in term 1.
I20260929 11:36:21.382546 20721 raft_consensus.cc:2454] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3656c32e4f049a5b91bd71c73d2c27f in term 1.
I20260929 11:36:21.383065 20789 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60; no voters:
I20260929 11:36:21.383368 21051 raft_consensus.cc:2781] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:21.383802 21051 raft_consensus.cc:686] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 1 LEADER]: Becoming Leader. State: Replica: a3656c32e4f049a5b91bd71c73d2c27f, State: Running, Role: LEADER
I20260929 11:36:21.384312 21051 consensus_queue.cc:237] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:21.388782 20581 catalog_manager.cc:5495] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f reported cstate change: term changed from 0 to 1, leader changed from <none> to a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194). New cstate: current_term: 1 leader_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } health_report { overall_health: UNKNOWN } } }
I20260929 11:36:21.421499 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.415326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.6.9.194:40547)
0929 11:36:21.415508 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.415513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.415615 (+ 102us) server_negotiation.cc:407] Connection header received
0929 11:36:21.415811 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.415815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.415898 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.416027 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:36:21.416426 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.417238 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.418914 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.419372 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.419524 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.420213 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:36:21.420857 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:36:21.420975 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.421119 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.421206 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:36:21.430780 20721 raft_consensus.cc:1261] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Refusing update from remote peer a3656c32e4f049a5b91bd71c73d2c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:21.430780 20984 raft_consensus.cc:1261] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Refusing update from remote peer a3656c32e4f049a5b91bd71c73d2c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:21.432001 21051 consensus_queue.cc:1035] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:21.432375 21056 consensus_queue.cc:1035] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:21.457357 21061 mvcc.cc:205] Tried to move back new op lower bound from 7334632576727871488 to 7334632576554344448. Current Snapshot: MvccSnapshot[applied={T|T < 7334632576727871488}]
I20260929 11:36:21.466472 20581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:21.466748 20581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:21.470680 20581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Sending DeleteTablet for 3 replicas of tablet 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:21.472049 20701 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:21 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:42860
I20260929 11:36:21.472023 20833 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:21 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:47344
I20260929 11:36:21.472157 20964 tablet_service.cc:1514] Processing DeleteTablet for tablet 370d3a42ed054d3c99abc0a3b0ad1068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:21 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39130
I20260929 11:36:21.472461 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20618
I20260929 11:36:21.473469 21070 tablet_replica.cc:330] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: stopping tablet replica
I20260929 11:36:21.474135 21070 raft_consensus.cc:2229] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:21.474747 21070 raft_consensus.cc:2258] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:21.477075 21070 ts_tablet_manager.cc:1905] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:21.484859 21070 ts_tablet_manager.cc:1918] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:21.485144 21070 log.cc:1196] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:21.485782 21070 ts_tablet_manager.cc:1939] T 370d3a42ed054d3c99abc0a3b0ad1068 P b7592e712f42470594fef5212e5e7d60: Deleting consensus metadata
I20260929 11:36:21.488057 20568 catalog_manager.cc:4841] TS b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): tablet 370d3a42ed054d3c99abc0a3b0ad1068 (table test-table [id=79b80185be0e451fa7fbee93c66ae409]) successfully deleted
I20260929 11:36:21.489034 21068 tablet_replica.cc:330] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: stopping tablet replica
I20260929 11:36:21.489578 21068 raft_consensus.cc:2229] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:21.490654 21068 raft_consensus.cc:2258] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:21.491588 21069 tablet_replica.cc:330] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: stopping tablet replica
I20260929 11:36:21.492130 21069 raft_consensus.cc:2229] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:36:21.492789 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.473317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.6.9.193:41087)
0929 11:36:21.473524 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.473531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.473590 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:36:21.473805 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.473811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.473902 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.474016 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:21.475820 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.477133 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.479256 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.480018 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.480169 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.483712 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:21.483737 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:21.483760 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:21.483813 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:21.488220 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.489313 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.489327 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.489331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.489533 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.490379 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.490385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.490390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.491046 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:21.491403 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.492104 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.492379 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:36:21.492944 21069 raft_consensus.cc:2258] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:21.493417 21068 ts_tablet_manager.cc:1905] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:21.494632 20761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:21.495092 21069 ts_tablet_manager.cc:1905] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:21.502074 21069 ts_tablet_manager.cc:1918] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:21.502310 21069 log.cc:1196] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:21.502950 21069 ts_tablet_manager.cc:1939] T 370d3a42ed054d3c99abc0a3b0ad1068 P a3656c32e4f049a5b91bd71c73d2c27f: Deleting consensus metadata
I20260929 11:36:21.505102 20566 catalog_manager.cc:4841] TS a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): tablet 370d3a42ed054d3c99abc0a3b0ad1068 (table test-table [id=79b80185be0e451fa7fbee93c66ae409]) successfully deleted
I20260929 11:36:21.590646 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20618
I20260929 11:36:21.606170 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20770
W20260929 11:36:21.606077 20568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.9.193:41087 (error 108)
I20260929 11:36:21.608104 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.607647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42886)
0929 11:36:21.607832 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.607961 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:36:21.608856 20568 catalog_manager.cc:4584] TS e3872a64de51485ea7dc5435e233efec (127.6.9.193:41087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 370d3a42ed054d3c99abc0a3b0ad1068: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
I20260929 11:36:21.623339 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.607341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.6.9.194:40547)
0929 11:36:21.607531 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.607537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.607557 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:36:21.607622 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.607626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.607701 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.607809 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:21.609653 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.610565 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.612563 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.613059 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.613188 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.616617 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:21.616632 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:21.616649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:21.616804 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:21.620182 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.621162 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.621171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.621173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.621360 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.621828 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.621834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.621836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.622235 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:21.622526 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.622848 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.623026 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:36:21.624167 20893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:21.662671 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.662143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42890)
0929 11:36:21.662367 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.662481 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:36:21.663921 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.663459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42900)
0929 11:36:21.663648 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.663749 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:36:21.724745 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.724184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42914)
0929 11:36:21.724419 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.724511 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 11:36:21.725934 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.725541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42926)
0929 11:36:21.725722 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.725804 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:36:21.736876 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20770
I20260929 11:36:21.752218 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20901
I20260929 11:36:21.766232 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.753021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.6.9.195:42991)
0929 11:36:21.753184 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:21.753188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:21.753209 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:36:21.753351 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:21.753355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:21.753422 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:21.753533 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:21.754906 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.755660 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.757107 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:21.757509 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:21.757624 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:21.760079 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:21.760094 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:21.760112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:21.760164 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:21.762913 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.763693 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.763701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.763704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.763900 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:21.764388 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:21.764393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:21.764396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:21.764903 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:21.765215 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:21.765591 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 11:36:21.765841 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:36:21.766955 21024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:21.806777 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.806193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42942)
0929 11:36:21.806444 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.806574 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:36:21.808079 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.807590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42958)
0929 11:36:21.807827 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.807922 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 11:36:21.857759 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20901
I20260929 11:36:21.871299 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 20548
I20260929 11:36:21.872495 20611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:21.987412 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:21.986679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:41087 (local address 127.0.0.1:42966)
0929 11:36:21.986864 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:21.986968 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 11:36:21.987411 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 20548
I20260929 11:36:22.006089 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--webserver_interface=127.6.9.254
--webserver_port=44587
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:22.337986 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:22.338302 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:22.338428 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:22.346035 21076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:22.346161 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:22.346225 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:22.346279 21076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:22.346326 21076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:22.356263 21076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=44587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:22.357623 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:22.358896 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:22.370210 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:22.370191 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:22.370769 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:22.371080 21076 server_base.cc:1034] running on GCE node
I20260929 11:36:22.371980 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:22.373462 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:22.374683 21076 hybrid_clock.cc:648] HybridClock initialized: now 1790681782374622 us; error 51 us; skew 500 ppm
I20260929 11:36:22.375018 21076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:22 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681782, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:22.378086 21076 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:22.379572 21076 webserver.cc:460] Webserver started at http://127.6.9.254:44587/ using document root <none> and password file <none>
I20260929 11:36:22.380254 21076 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:22.380378 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:22.386193 21076 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 11:36:22.389403 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:22.391036 21076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:36:22.391240 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7a6a580c89b6434faf65b3e22397835b"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "8c77bad08c640ded85308e56a0230292"
server_key_iv: "fb8833bc641ff3a1eedff3f497353e82"
server_key_version: "encryptionkey@0"
I20260929 11:36:22.392333 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:22.419657 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:22.424435 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:22.424808 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:22.444737 21076 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40885
I20260929 11:36:22.444784 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40885 every 8 connection(s)
I20260929 11:36:22.446497 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:22.448896 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21076
I20260929 11:36:22.449654 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:41087
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=44105
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:22.453994 21143 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:36:22.457604 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap starting.
I20260929 11:36:22.478926 21143 log.cc:824] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:22.495371 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:36:22.495922 21143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b: Bootstrap complete.
I20260929 11:36:22.506559 21143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:22.508049 21143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Initialized, Role: FOLLOWER
I20260929 11:36:22.508972 21143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:22.509295 21143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:22.509440 21143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:22.509629 21143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:36:22.515053 21143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:22.515622 21143 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6a580c89b6434faf65b3e22397835b; no voters:
I20260929 11:36:22.516269 21143 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:36:22.516425 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:36:22.518479 21143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:22.520915 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [term 3 LEADER]: Becoming Leader. State: Replica: 7a6a580c89b6434faf65b3e22397835b, State: Running, Role: LEADER
I20260929 11:36:22.521677 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } }
I20260929 11:36:22.524225 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6a580c89b6434faf65b3e22397835b. Latest consensus state: current_term: 3 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:22.524609 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:22.524886 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7a6a580c89b6434faf65b3e22397835b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6a580c89b6434faf65b3e22397835b" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40885 } } }
I20260929 11:36:22.525425 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6a580c89b6434faf65b3e22397835b [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:22.533885 21156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:22.543116 21156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:22.545034 21156 catalog_manager.cc:1259] Loaded cluster ID: ceea1545804d4a768be97a9c4a9ce1f7
I20260929 11:36:22.545243 21156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 11:36:22.547668 21156 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5affbf79ad google::LogMessage::Fail() at ??:0
@ 0x7f5affbfa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5affbf74de google::LogMessage::Flush() at ??:0
@ 0x7f5affbf8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5b1562a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5b15627976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5b156af610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5b156af439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5b02843345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5b0284691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5b02846799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5b02821eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5b05afb6db start_thread at ??:0
@ 0x7f5afea9161f clone at ??:0
W20260929 11:36:22.866305 21145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:22.866616 21145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:22.866722 21145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:22.874070 21145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:22.874191 21145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:22.874316 21145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:22.884344 21145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:41087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=44105
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:22.885706 21145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:22.887018 21145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:22.900697 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:22.900753 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:22.900724 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:22.901718 21145 server_base.cc:1034] running on GCE node
I20260929 11:36:22.902377 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:22.903436 21145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:22.904640 21145 hybrid_clock.cc:648] HybridClock initialized: now 1790681782904586 us; error 52 us; skew 500 ppm
I20260929 11:36:22.904984 21145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:22 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681782, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:22.908088 21145 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:22.909623 21145 webserver.cc:460] Webserver started at http://127.6.9.193:44105/ using document root <none> and password file <none>
I20260929 11:36:22.910370 21145 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:22.910496 21145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:22.916050 21145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:36:22.919234 21195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:22.920802 21145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:36:22.920974 21145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "e3872a64de51485ea7dc5435e233efec"
format_stamp: "Formatted at 2026-09-29 11:36:16 on dist-test-slave-c0l9"
server_key: "d5ab63b913258d763a1ab2b5e98f1bbb"
server_key_iv: "dd2f4a0ce4cc6b204d67b8f54d0be1b3"
server_key_version: "encryptionkey@0"
I20260929 11:36:22.922000 21145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:22.935652 21145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:22.940024 21145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:22.940307 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:22.941399 21145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:22.943392 21202 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 11:36:22.949719 21145 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 11:36:22.949895 21145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 11:36:22.950027 21145 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 11:36:22.953382 21145 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 11:36:22.953477 21145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:22.953846 21202 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Bootstrap starting.
I20260929 11:36:22.962984 21202 log.cc:824] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:22.981882 21202 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:36:22.982731 21202 tablet_bootstrap.cc:492] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Bootstrap complete.
I20260929 11:36:22.983557 21202 ts_tablet_manager.cc:1397] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260929 11:36:22.988770 21145 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:41087
I20260929 11:36:22.988830 21309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:41087 every 8 connection(s)
I20260929 11:36:22.990438 21145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:22.994143 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21145
I20260929 11:36:22.995074 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:40547
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=44477
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:22.993994 21202 raft_consensus.cc:348] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:22.995500 21202 raft_consensus.cc:729] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3872a64de51485ea7dc5435e233efec, State: Initialized, Role: FOLLOWER
I20260929 11:36:22.996225 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:22.993446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:40585)
0929 11:36:22.994651 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:22.994963 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":572,"threads_started":1}
I20260929 11:36:22.996205 21202 consensus_queue.cc:260] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
W20260929 11:36:22.996979 21310 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:40885 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:40885: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
I20260929 11:36:22.998049 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:22.997608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:37049)
0929 11:36:22.997829 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:22.997911 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 11:36:22.998512 21202 ts_tablet_manager.cc:1428] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 11:36:22.999051 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:22.998692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:35815)
0929 11:36:22.998852 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:22.998930 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:36:22.999277 21310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:36:22.999907 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:22.999618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:40101)
0929 11:36:22.999748 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:22.999813 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 11:36:23.328536 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:23.328882 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:23.329010 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:23.336375 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:23.336509 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:23.336691 21314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:23.346794 21314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:40547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=44477
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:23.348155 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:23.349578 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:23.363044 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:23.363001 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:23.363546 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:23.364137 21314 server_base.cc:1034] running on GCE node
I20260929 11:36:23.364841 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:23.365859 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:23.367074 21314 hybrid_clock.cc:648] HybridClock initialized: now 1790681783367018 us; error 54 us; skew 500 ppm
I20260929 11:36:23.367415 21314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:23 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681783, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:23.370635 21314 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:23.372160 21314 webserver.cc:460] Webserver started at http://127.6.9.194:44477/ using document root <none> and password file <none>
I20260929 11:36:23.372967 21314 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:23.373087 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:23.378759 21314 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:36:23.381609 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.383066 21314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:23.383241 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "a3656c32e4f049a5b91bd71c73d2c27f"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "475934661d1c00a9090196fb45b3707f"
server_key_iv: "46358439c2884f6a58d701fb0ad9a938"
server_key_version: "encryptionkey@0"
I20260929 11:36:23.384248 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:23.414467 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:23.419252 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:23.419541 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:23.420642 21314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:23.422268 21314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:23.422372 21314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.422482 21314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:23.422557 21314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.454811 21314 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:40547
I20260929 11:36:23.454869 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:40547 every 8 connection(s)
I20260929 11:36:23.456243 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:23.458487 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21314
I20260929 11:36:23.459211 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:42991
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:40885
--webserver_port=35811
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:36:23.461302 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.459400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:60559)
0929 11:36:23.460175 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.460369 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":285,"threads_started":1}
W20260929 11:36:23.462013 21442 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:40885 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:40885: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
I20260929 11:36:23.462872 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.462534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:51227)
0929 11:36:23.462693 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.462771 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 11:36:23.463737 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.463390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:40217)
0929 11:36:23.463529 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.463624 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 11:36:23.463953 21442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:36:23.785012 21445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:23.785331 21445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:23.785451 21445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:23.793249 21445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:23.793390 21445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:23.793529 21445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:23.803918 21445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:42991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=35811
--tserver_master_addrs=127.6.9.254:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:23.805343 21445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:23.806968 21445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:23.822129 21450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:23.822225 21453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:23.822363 21451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:23.823199 21445 server_base.cc:1034] running on GCE node
I20260929 11:36:23.823927 21445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:23.825047 21445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:23.826258 21445 hybrid_clock.cc:648] HybridClock initialized: now 1790681783826189 us; error 69 us; skew 500 ppm
I20260929 11:36:23.826653 21445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:23 dist-test-slave-c0l9 krb5kdc[19990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681783, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:23.829964 21445 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:23.831586 21445 webserver.cc:460] Webserver started at http://127.6.9.195:35811/ using document root <none> and password file <none>
I20260929 11:36:23.832305 21445 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:23.832427 21445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:23.838045 21445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:36:23.840942 21460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.842415 21445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:23.842581 21445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b7592e712f42470594fef5212e5e7d60"
format_stamp: "Formatted at 2026-09-29 11:36:17 on dist-test-slave-c0l9"
server_key: "15736c85c1193725d75403d543b5f412"
server_key_iv: "90afc33a92a36f17a8067b917cac82b1"
server_key_version: "encryptionkey@0"
I20260929 11:36:23.843595 21445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:23.892500 21445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:23.897066 21445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:23.897320 21445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:23.898330 21445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:23.899658 21445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:23.899770 21445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.899878 21445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:23.899950 21445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:23.929865 21445 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:42991
I20260929 11:36:23.929916 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:42991 every 8 connection(s)
I20260929 11:36:23.931308 21445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:23.933476 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21445
I20260929 11:36:23.937597 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.934811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:53763)
0929 11:36:23.936425 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.936640 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1358,"thread_start_us":1132,"threads_started":1}
W20260929 11:36:23.938313 21573 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:40885 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:40885: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
I20260929 11:36:23.939158 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.938807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:52133)
0929 11:36:23.938958 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.939027 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 11:36:23.940076 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:23.939730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:60721)
0929 11:36:23.939885 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:23.939961 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:36:23.940402 21573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:36:24.001750 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.000845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:47749)
0929 11:36:24.001419 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:24.001538 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":312,"threads_started":1}
I20260929 11:36:24.377440 21578 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:24.377758 21578 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:24.379621 21578 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
Sep 29 11:36:24 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681782, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
Sep 29 11:36:24 dist-test-slave-c0l9 krb5kdc[19990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681782, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:36:24.402290 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.379908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:42991 (local address 127.6.9.193:60999)
0929 11:36:24.380176 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:24.380223 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:24.380849 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:24.384688 (+ 3839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:24.384697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:24.385508 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:24.386396 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:24.386421 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.388023 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.388027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:24.389554 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:24.389568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.390827 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.390834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:24.391112 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:24.391935 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:24.391953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:24.394545 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:24.398243 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:24.398253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:24.398268 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:24.398834 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:24.399278 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:24.399283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:24.399287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:24.399500 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:24.400291 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:24.400299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:24.400830 (+ 531us) client_negotiation.cc:769] Sending connection context
0929 11:36:24.401195 (+ 365us) client_negotiation.cc:240] Negotiation successful
0929 11:36:24.401940 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":456,"spinlock_wait_cycles":41984}
I20260929 11:36:24.402298 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.379965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:40547 (local address 127.6.9.193:58553)
0929 11:36:24.381899 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:24.381915 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:24.382053 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:24.384654 (+ 2601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:24.384696 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:24.385508 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:24.386398 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:24.386421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.388031 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.388035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:24.389558 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:24.389572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.390801 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.390807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:24.391115 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:24.391883 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:24.391918 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:24.394708 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:24.398243 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:24.398253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:24.398267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:24.398834 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:24.399291 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:24.399297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:24.399300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:24.399509 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:24.400291 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:24.400299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:24.400840 (+ 541us) client_negotiation.cc:769] Sending connection context
0929 11:36:24.401195 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 11:36:24.401797 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"mutex_wait_us":294,"spinlock_wait_cycles":67328,"thread_start_us":1655,"threads_started":1}
I20260929 11:36:24.402580 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.380269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:60999 (local address 127.6.9.195:42991)
0929 11:36:24.382197 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:24.382206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:24.382245 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:36:24.383808 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:24.383817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:24.384165 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:24.384471 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:24.386620 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.387831 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.389704 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.390583 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.390758 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:24.394917 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:24.394926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:24.394942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:24.394986 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:24.398067 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:24.398987 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:24.398995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:24.399000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:24.399171 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:24.399614 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:24.399618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:24.399620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:24.400161 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:24.400346 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:24.401883 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0929 11:36:24.402221 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":1641,"threads_started":1}
I20260929 11:36:24.404542 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.380204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58553 (local address 127.6.9.194:40547)
0929 11:36:24.381669 (+ 1465us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:24.381689 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:24.382050 (+ 361us) server_negotiation.cc:407] Connection header received
0929 11:36:24.383681 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:24.383691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:24.384115 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:24.384458 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:24.386626 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.387833 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.389729 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:24.390578 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:24.390755 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:24.395008 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:24.395017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:24.395031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:24.395072 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:24.398069 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:24.398989 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:24.398996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:24.399001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:24.399179 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:24.399624 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:24.399628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:24.399630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:24.400151 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:24.400336 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:24.403903 (+ 3567us) server_negotiation.cc:299] Negotiation successful
0929 11:36:24.404245 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":1100,"threads_started":1}
W20260929 11:36:24.405810 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:24.407102 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:24.407248 21197 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:24.407675 21578 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:24.465514 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.464574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:57021)
0929 11:36:24.465122 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:24.465263 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":267,"threads_started":1}
I20260929 11:36:24.941828 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:24.941039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:33929)
0929 11:36:24.941487 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:24.941612 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260929 11:36:25.003392 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:25.002662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:48217)
0929 11:36:25.003124 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:25.003216 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":264,"threads_started":1}
I20260929 11:36:25.467422 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:25.466626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:47327)
0929 11:36:25.467155 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:25.467255 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260929 11:36:25.943694 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:25.942781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:50799)
0929 11:36:25.943342 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:25.943463 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":253,"threads_started":1}
I20260929 11:36:26.005151 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:26.004328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:37193)
0929 11:36:26.004856 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:26.004950 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":231,"threads_started":1}
I20260929 11:36:26.130726 21588 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:26.130954 21588 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:26.131639 21588 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
W20260929 11:36:26.132762 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:26.132956 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:26.133055 21197 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:26.133311 21588 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:26.469101 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:26.468339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:49255)
0929 11:36:26.468841 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:26.468926 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
I20260929 11:36:26.945616 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:26.944728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:47307)
0929 11:36:26.945241 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:26.945374 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
I20260929 11:36:27.007054 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:27.006216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:33721)
0929 11:36:27.006749 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:27.006850 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":257,"threads_started":1}
I20260929 11:36:27.471033 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:27.470210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:58769)
0929 11:36:27.470741 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:27.470839 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260929 11:36:27.947651 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:27.946677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:56241)
0929 11:36:27.947289 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:27.947418 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260929 11:36:28.009195 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:28.008341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:33167)
0929 11:36:28.008894 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:28.008999 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":263,"threads_started":1}
I20260929 11:36:28.236913 21595 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:28.237159 21595 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:28.238049 21595 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
W20260929 11:36:28.238984 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:28.239174 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:28.239285 21197 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:28.239529 21595 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:28.472863 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:28.472073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:51029)
0929 11:36:28.472568 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:28.472655 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260929 11:36:28.949502 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:28.948605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:53497)
0929 11:36:28.949172 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:28.949293 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":316,"threads_started":1}
I20260929 11:36:29.010993 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:29.010197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:36901)
0929 11:36:29.010710 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:29.010802 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
I20260929 11:36:29.474663 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:29.473809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:45647)
0929 11:36:29.474373 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:29.474460 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":326,"threads_started":1}
I20260929 11:36:29.825340 21600 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:29.825600 21600 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:29.826347 21600 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
W20260929 11:36:29.827364 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:29.827572 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:29.827699 21199 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:29.827966 21600 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:29.951426 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:29.950520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:33289)
0929 11:36:29.951058 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:29.951168 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
I20260929 11:36:30.012918 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:30.012034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:34099)
0929 11:36:30.012565 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:30.012688 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":259,"threads_started":1}
I20260929 11:36:30.476591 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:30.475720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:36131)
0929 11:36:30.476293 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:30.476394 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":256,"threads_started":1}
I20260929 11:36:30.953284 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:30.952416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:47133)
0929 11:36:30.952967 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:30.953055 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":261,"threads_started":1}
I20260929 11:36:31.014901 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:31.014050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:58385)
0929 11:36:31.014596 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:31.014690 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260929 11:36:31.478587 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:31.477727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:43779)
0929 11:36:31.478279 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:31.478388 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":308,"threads_started":1}
I20260929 11:36:31.955086 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:31.954325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:34055)
0929 11:36:31.954809 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:31.954909 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260929 11:36:32.016947 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:32.016071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:45645)
0929 11:36:32.016586 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:32.016732 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260929 11:36:32.480466 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:32.479599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:38583)
0929 11:36:32.480184 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:32.480282 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":288,"threads_started":1}
I20260929 11:36:32.957156 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:32.956188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:33055)
0929 11:36:32.956849 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:32.956952 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":281,"threads_started":1}
I20260929 11:36:33.019119 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:33.018299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:49047)
0929 11:36:33.018835 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:33.018928 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260929 11:36:33.482406 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:33.481592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:46507)
0929 11:36:33.482095 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:33.482186 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260929 11:36:33.959143 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:33.958301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:47163)
0929 11:36:33.958862 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:33.958966 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260929 11:36:34.021004 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:34.020132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:46943)
0929 11:36:34.020703 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:34.020836 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":298,"threads_started":1}
I20260929 11:36:34.484313 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:34.483502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:59289)
0929 11:36:34.484029 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:34.484127 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":285,"threads_started":1}
I20260929 11:36:34.508894 21616 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:34.509093 21616 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:34.509786 21616 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
W20260929 11:36:34.510851 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:34.511023 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:34.511228 21197 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:34.511489 21616 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:34.961058 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:34.960288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:43065)
0929 11:36:34.960811 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:34.960897 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":247,"threads_started":1}
I20260929 11:36:35.022768 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:35.021970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:42387)
0929 11:36:35.022478 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:35.022577 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":230,"threads_started":1}
I20260929 11:36:35.486230 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:35.485392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:45187)
0929 11:36:35.485925 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:35.486019 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":270,"threads_started":1}
I20260929 11:36:35.962745 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:35.962005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:35235)
0929 11:36:35.962460 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:35.962547 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260929 11:36:36.024705 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:36.023848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:47345)
0929 11:36:36.024394 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:36.024498 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":280,"threads_started":1}
I20260929 11:36:36.488097 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:36.487362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:39661)
0929 11:36:36.487841 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:36.487920 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
I20260929 11:36:36.964531 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:36.963755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:47397)
0929 11:36:36.964265 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:36.964356 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260929 11:36:37.026625 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:37.025702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:59687)
0929 11:36:37.026329 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:37.026420 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
I20260929 11:36:37.489835 21625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:37.489058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:58249)
0929 11:36:37.489536 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:37.489630 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260929 11:36:37.966253 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:37.965500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:44027)
0929 11:36:37.965934 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:37.966027 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
I20260929 11:36:38.028523 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:38.027811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:51901)
0929 11:36:38.028267 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:38.028359 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20260929 11:36:38.491849 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:38.491061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.194:40929)
0929 11:36:38.491562 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:38.491660 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260929 11:36:38.734544 21629 raft_consensus.cc:482] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:38.734797 21629 raft_consensus.cc:504] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3656c32e4f049a5b91bd71c73d2c27f" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 40547 } } peers { permanent_uuid: "e3872a64de51485ea7dc5435e233efec" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 41087 } } peers { permanent_uuid: "b7592e712f42470594fef5212e5e7d60" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 42991 } }
I20260929 11:36:38.735479 21629 leader_election.cc:288] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547), b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991)
W20260929 11:36:38.736393 21197 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3656c32e4f049a5b91bd71c73d2c27f (127.6.9.194:40547): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
W20260929 11:36:38.736573 21199 leader_election.cc:341] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7592e712f42470594fef5212e5e7d60 (127.6.9.195:42991): Not found: Tablet not found: 370d3a42ed054d3c99abc0a3b0ad1068
I20260929 11:36:38.736732 21199 leader_election.cc:302] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3872a64de51485ea7dc5435e233efec; no voters: a3656c32e4f049a5b91bd71c73d2c27f, b7592e712f42470594fef5212e5e7d60
I20260929 11:36:38.737000 21629 raft_consensus.cc:2726] T 370d3a42ed054d3c99abc0a3b0ad1068 P e3872a64de51485ea7dc5435e233efec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:36:38.938777 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21145
I20260929 11:36:38.954762 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:38.939851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.6.9.193:41087)
0929 11:36:38.940522 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:38.940532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:38.940563 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:36:38.940634 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:38.940642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:38.940972 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:38.941145 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:38.943376 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:38.944454 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:38.945763 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:38.946324 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:38.946423 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:38.949008 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:38.949021 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:38.949037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:38.949073 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:38.951932 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:38.952626 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:38.952634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:38.952636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:38.952831 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:38.953178 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:38.953182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:38.953184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:38.953559 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:38.953829 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:38.954243 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 11:36:38.954463 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":368,"threads_started":1}
I20260929 11:36:38.956050 21305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:38.968335 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:38.967561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.195:54343)
0929 11:36:38.968071 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:38.968162 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260929 11:36:39.057530 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:39.048025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:40885 (local address 127.6.9.193:36631)
0929 11:36:39.057059 (+ 9034us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:39.057232 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8893,"thread_start_us":8749,"threads_started":1}
I20260929 11:36:39.057499 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21145
I20260929 11:36:39.072710 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21314
I20260929 11:36:39.086514 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:39.073427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.6.9.194:40547)
0929 11:36:39.073878 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:39.073882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:39.073900 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:36:39.073960 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:39.073963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:39.074025 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:39.074122 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:39.075498 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:39.076286 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:39.077681 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:39.078118 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:39.078227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:39.080870 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:39.080877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:39.080881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:39.080920 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:39.083489 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:39.084259 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:39.084264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:39.084267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:39.084431 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:39.085005 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:39.085010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:39.085012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:39.085318 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:39.085541 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:39.085924 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 11:36:39.086122 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260929 11:36:39.087312 21437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:39.172132 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21314
I20260929 11:36:39.186642 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21445
I20260929 11:36:39.200639 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:39.187495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.6.9.195:42991)
0929 11:36:39.188145 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:39.188150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:39.188167 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:36:39.188239 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:39.188243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:39.188304 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:39.188394 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:39.189828 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:39.190625 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:39.191859 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:39.192335 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:39.192428 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:39.195254 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:39.195259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:39.195263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:39.195300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:39.197999 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:39.198750 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:39.198755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:39.198757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:39.198903 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:39.199340 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:39.199344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:39.199346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:39.199656 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:39.199893 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:39.200103 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 11:36:39.200283 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":358,"threads_started":1}
I20260929 11:36:39.201484 21568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:39.285271 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21445
2026-09-29T11:36:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24582 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:36:39 dist-test-slave-c0l9 krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:36:39 dist-test-slave-c0l9 krb5kdc[21640](info): set up 2 sockets
Sep 29 11:36:39 dist-test-slave-c0l9 krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260929 11:36:40.309082 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:36:40 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:36:40.563645 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:40.915632 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:40.915956 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:40.916088 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:40.923359 21653 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:36:40.923484 21653 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:36:40.923545 21653 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:36:40.923617 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:36:40.923695 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:40.923799 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:40.923856 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:36:40.923938 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:36:40.934082 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:40.935442 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:40.936831 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:40.949003 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:40.948964 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:40.949205 21653 server_base.cc:1034] running on GCE node
W20260929 11:36:40.949043 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:40.950510 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:40.952018 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:40.953256 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790681800953189 us; error 43 us; skew 500 ppm
I20260929 11:36:40.953591 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:40 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:40.962669 21653 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:36:40.964473 21653 webserver.cc:460] Webserver started at http://127.6.9.254:39665/ using document root <none> and password file <none>
I20260929 11:36:40.965358 21653 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:40.965466 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:40.965812 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:40.969103 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "c8cded25f3314a47b7b5a05e5530c464"
format_stamp: "Formatted at 2026-09-29 11:36:40 on dist-test-slave-c0l9"
server_key: "b9063dab6e2a7d46e496ae067ae8bcec"
server_key_iv: "d35ff53d302d176271a3a5240bf40289"
server_key_version: "encryptionkey@0"
I20260929 11:36:40.969950 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "c8cded25f3314a47b7b5a05e5530c464"
format_stamp: "Formatted at 2026-09-29 11:36:40 on dist-test-slave-c0l9"
server_key: "b9063dab6e2a7d46e496ae067ae8bcec"
server_key_iv: "d35ff53d302d176271a3a5240bf40289"
server_key_version: "encryptionkey@0"
I20260929 11:36:40.975529 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:36:40.979653 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:40.981387 21653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:36:40.981573 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c8cded25f3314a47b7b5a05e5530c464"
format_stamp: "Formatted at 2026-09-29 11:36:40 on dist-test-slave-c0l9"
server_key: "b9063dab6e2a7d46e496ae067ae8bcec"
server_key_iv: "d35ff53d302d176271a3a5240bf40289"
server_key_version: "encryptionkey@0"
I20260929 11:36:40.981812 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:41.001535 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:41.006385 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:41.006664 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:41.025756 21653 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46109
I20260929 11:36:41.025779 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46109 every 8 connection(s)
I20260929 11:36:41.027421 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:36:41.031755 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:41.036859 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21653
I20260929 11:36:41.037072 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:36:41.037730 6183 external_mini_cluster.cc:1428] Setting key 932c17814400576ccebc842c50c296c6
I20260929 11:36:41.044600 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap starting.
I20260929 11:36:41.049000 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:41.050197 21720 log.cc:824] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:41.053618 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: No bootstrap required, opened a new log
Sep 29 11:36:41 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:41.064096 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:36:41.064716 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:41.064848 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Initialized, Role: FOLLOWER
I20260929 11:36:41.065724 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:36:41.066053 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:36:41.066207 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:36:41.066398 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:41.066632 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:41.039919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.6.9.254:46109)
0929 11:36:41.040872 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:41.040889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:41.040952 (+ 63us) server_negotiation.cc:407] Connection header received
0929 11:36:41.043443 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:41.043479 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:41.044069 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:41.044609 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:41.046223 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.048404 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.050233 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.051292 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.051458 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:41.056185 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:41.056198 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:41.056222 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:41.056287 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:41.060773 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:41.061759 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:41.061770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:41.061781 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:41.062061 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:41.062605 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:41.062610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:41.062614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:41.063490 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:41.063830 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:41.064381 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 11:36:41.064976 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":360,"threads_started":1}
I20260929 11:36:41.069502 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:36:41.070109 21720 leader_election.cc:302] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8cded25f3314a47b7b5a05e5530c464; no voters:
I20260929 11:36:41.070725 21720 leader_election.cc:288] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:36:41.070858 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:41.071570 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 LEADER]: Becoming Leader. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Running, Role: LEADER
I20260929 11:36:41.072129 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:36:41.072997 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:36:41.075625 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8cded25f3314a47b7b5a05e5530c464. Latest consensus state: current_term: 1 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:36:41.075623 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:36:41.076702 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:41.076869 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:36:41.077625 21732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:36:41.083395 21732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:36:41.095571 21732 catalog_manager.cc:1347] Generated new cluster ID: eea8e8f5b07b4ccca60678980746a7d8
I20260929 11:36:41.095718 21732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:36:41.105006 21732 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:36:41.106953 21732 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:36:41.122093 21732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Generated new TSK 0
I20260929 11:36:41.123019 21732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:36:41.182507 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:41.536273 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:41.536728 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:41.536885 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:41.544579 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:41.544741 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:41.544904 21746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:36:41.555117 21746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:41.556458 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:41.557861 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:41.570327 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:41.570356 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:41.570449 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:41.571113 21746 server_base.cc:1034] running on GCE node
I20260929 11:36:41.571734 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:41.572636 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:41.573880 21746 hybrid_clock.cc:648] HybridClock initialized: now 1790681801573820 us; error 45 us; skew 500 ppm
I20260929 11:36:41.574218 21746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:41 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681801, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:41.582185 21746 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:36:41.583791 21746 webserver.cc:460] Webserver started at http://127.6.9.193:39295/ using document root <none> and password file <none>
I20260929 11:36:41.584487 21746 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:41.584609 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:41.584955 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:41.588192 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "dc50dbec2bbb46afae886bb342cc1017"
format_stamp: "Formatted at 2026-09-29 11:36:41 on dist-test-slave-c0l9"
server_key: "e76826240805893b5473ab7299188609"
server_key_iv: "45d638d02d0f0002a5dbbf4c382890d4"
server_key_version: "encryptionkey@0"
I20260929 11:36:41.589073 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "dc50dbec2bbb46afae886bb342cc1017"
format_stamp: "Formatted at 2026-09-29 11:36:41 on dist-test-slave-c0l9"
server_key: "e76826240805893b5473ab7299188609"
server_key_iv: "45d638d02d0f0002a5dbbf4c382890d4"
server_key_version: "encryptionkey@0"
I20260929 11:36:41.594957 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:36:41.599014 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:41.600775 21746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:41.600972 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "dc50dbec2bbb46afae886bb342cc1017"
format_stamp: "Formatted at 2026-09-29 11:36:41 on dist-test-slave-c0l9"
server_key: "e76826240805893b5473ab7299188609"
server_key_iv: "45d638d02d0f0002a5dbbf4c382890d4"
server_key_version: "encryptionkey@0"
I20260929 11:36:41.601199 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:41.617348 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:41.622081 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:41.622385 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:41.623391 21746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:41.624871 21746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:41.624951 21746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:41.625069 21746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:41.625128 21746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:41.658134 21746 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:36069
I20260929 11:36:41.658191 21873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:36069 every 8 connection(s)
I20260929 11:36:41.659551 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:36:41.669168 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21746
I20260929 11:36:41.669392 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:36:41.670078 6183 external_mini_cluster.cc:1428] Setting key cd420c0e222fa3117e598158b332ac23
Sep 29 11:36:41 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681801, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:41.682350 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:41.662391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:60427 (local address 127.6.9.254:46109)
0929 11:36:41.662942 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:41.662948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:41.663989 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 11:36:41.665995 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:41.666000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:41.666080 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:41.666192 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:41.667871 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.668636 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.670768 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.671281 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.671424 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:41.675803 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:41.675809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:41.675813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:41.675857 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:41.678774 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:41.679700 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:41.679705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:41.679708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:41.679854 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:41.680348 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:41.680352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:41.680355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:41.680691 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:41.680855 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:41.681686 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 11:36:41.681960 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":254,"threads_started":1}
I20260929 11:36:41.683429 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:41.663004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:60427)
0929 11:36:41.663815 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:41.663862 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:41.665625 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:41.666422 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:41.666432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:41.667009 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:41.667638 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:41.667659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.668876 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.668880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:41.670570 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:41.670589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:41.671555 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:41.671562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:41.671807 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:41.672656 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:41.672732 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:41.675410 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:41.678995 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:41.679004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:41.679021 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:41.679542 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:41.679985 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:41.679990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:41.679992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:41.680221 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:41.680848 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:41.680857 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:41.681265 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 11:36:41.681554 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 11:36:41.682195 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":287,"threads_started":1}
I20260929 11:36:41.685359 21874 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:36:41.685698 21874 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:41.686566 21874 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
I20260929 11:36:41.689312 21685 ts_manager.cc:194] Registered new tserver with Master: dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069)
I20260929 11:36:41.691586 21685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:60427
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:36:41.722302 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:42.062136 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:42.062471 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:42.062595 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:42.070519 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:42.070696 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:42.070847 21880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:36:42.081252 21880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:42.082650 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:42.084077 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:42.098601 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:42.098582 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:42.099202 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:42.100250 21880 server_base.cc:1034] running on GCE node
I20260929 11:36:42.101058 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:42.102141 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:42.103361 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790681802103301 us; error 50 us; skew 500 ppm
I20260929 11:36:42.103698 21880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:42 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681802, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:42.107029 21880 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:36:42.108551 21880 webserver.cc:460] Webserver started at http://127.6.9.194:37525/ using document root <none> and password file <none>
I20260929 11:36:42.109277 21880 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:42.109388 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:42.109663 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:42.112706 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "7170db9d4e8b4f0eab6d14785859b8e4"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "4001d537f2a0d2a563c93013d9aa6c1d"
server_key_iv: "f7847a739b54c6bfc55651cebf5ccb81"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.113462 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "7170db9d4e8b4f0eab6d14785859b8e4"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "4001d537f2a0d2a563c93013d9aa6c1d"
server_key_iv: "f7847a739b54c6bfc55651cebf5ccb81"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.118717 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:36:42.122283 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.123648 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:36:42.123811 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "7170db9d4e8b4f0eab6d14785859b8e4"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "4001d537f2a0d2a563c93013d9aa6c1d"
server_key_iv: "f7847a739b54c6bfc55651cebf5ccb81"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.123996 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:42.144331 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:42.148749 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:42.149019 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:42.150014 21880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:42.151391 21880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:42.151466 21880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.151573 21880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:42.151630 21880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.182749 21880 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42599
I20260929 11:36:42.182832 22007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42599 every 8 connection(s)
I20260929 11:36:42.184089 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:36:42.185761 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 21880
I20260929 11:36:42.185932 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:36:42.186457 6183 external_mini_cluster.cc:1428] Setting key 6a2bff1dd88af88f49e31a39f3804637
Sep 29 11:36:42 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681802, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:42.205710 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.186932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:58195 (local address 127.6.9.254:46109)
0929 11:36:42.187375 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.187381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.188945 (+ 1564us) server_negotiation.cc:407] Connection header received
0929 11:36:42.190907 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.190912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.191002 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.191104 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:42.192865 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.193852 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.195583 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.196013 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.196135 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.199791 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:42.199796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:42.199800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:42.199833 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:42.202390 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.203180 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.203184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.203187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.203316 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.203756 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.203760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.203762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.204105 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:42.204246 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.205046 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.205323 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260929 11:36:42.206777 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.187950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:58195)
0929 11:36:42.188782 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.188829 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.190571 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.191342 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.191355 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.191974 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:42.192616 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.192640 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.194063 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.194068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.195394 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.195409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.196266 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.196274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.196503 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.197203 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:42.197229 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:42.199538 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:42.202553 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:42.202563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:42.202582 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:42.203040 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:42.203426 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:42.203431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:42.203434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:42.203636 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:42.204231 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:42.204238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:42.204600 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.204895 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.205524 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":322,"threads_started":1}
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:36:42.208336 22008 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:36:42.208653 22008 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:42.209462 22008 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
I20260929 11:36:42.211229 21685 ts_manager.cc:194] Registered new tserver with Master: 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599)
I20260929 11:36:42.212087 21685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:58195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:36:42.240509 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:36:42.578281 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:36:42.578595 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:36:42.578702 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:36:42.586166 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:36:42.586294 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:36:42.586417 22014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:36:42.597864 22014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:36:42.599340 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:36:42.600754 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:36:42.616420 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:42.616889 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:36:42.616935 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:36:42.617905 22014 server_base.cc:1034] running on GCE node
I20260929 11:36:42.618597 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:36:42.619601 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:36:42.620816 22014 hybrid_clock.cc:648] HybridClock initialized: now 1790681802620757 us; error 52 us; skew 500 ppm
I20260929 11:36:42.621138 22014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:36:42 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681802, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:36:42.624218 22014 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:36:42.625721 22014 webserver.cc:460] Webserver started at http://127.6.9.195:36397/ using document root <none> and password file <none>
I20260929 11:36:42.626427 22014 fs_manager.cc:359] Metadata directory not provided
I20260929 11:36:42.626550 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:36:42.626865 22014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:36:42.630096 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "b01ec4485fad48de94a264113b699930"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "3a7e58606bc2a2f3ee4196374493f35f"
server_key_iv: "b91c631d5dcbcd056db1e09e5bdf4338"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.630913 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "b01ec4485fad48de94a264113b699930"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "3a7e58606bc2a2f3ee4196374493f35f"
server_key_iv: "b91c631d5dcbcd056db1e09e5bdf4338"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.636379 22014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:36:42.640360 22029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.641899 22014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:36:42.642083 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b01ec4485fad48de94a264113b699930"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "3a7e58606bc2a2f3ee4196374493f35f"
server_key_iv: "b91c631d5dcbcd056db1e09e5bdf4338"
server_key_version: "encryptionkey@0"
I20260929 11:36:42.642298 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:36:42.658057 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:36:42.663080 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:36:42.663383 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:36:42.664392 22014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:36:42.665860 22014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:36:42.665946 22014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.666066 22014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:36:42.666127 22014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:36:42.695304 21874 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:36:42.697994 22014 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:37909
I20260929 11:36:42.698027 22141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:37909 every 8 connection(s)
I20260929 11:36:42.699256 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:36:42.704013 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22014
I20260929 11:36:42.704180 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:36:42.704607 6183 external_mini_cluster.cc:1428] Setting key 1054724a41e888d9c46bbc1d6eb9d975
Sep 29 11:36:42 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681802, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:36:42.720497 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.701764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:56831 (local address 127.6.9.254:46109)
0929 11:36:42.701964 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.701970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.703190 (+ 1220us) server_negotiation.cc:407] Connection header received
0929 11:36:42.705418 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.705424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.705545 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.705675 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:42.707867 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.708856 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.710478 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.710927 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.711052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.714565 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:42.714571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:42.714574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:42.714610 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:42.717252 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.717989 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.717995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.717998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.718130 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.718526 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.718531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.718533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.718907 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:42.719031 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.719911 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.720165 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:36:42.721563 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.702293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:56831)
0929 11:36:42.703020 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.703057 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.705027 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.706010 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.706020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.706789 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:36:42.707625 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.707667 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.708995 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.708999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.710305 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.710319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.711126 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.711131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.711345 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.712031 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:36:42.712056 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:36:42.714306 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:36:42.717395 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:42.717402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:42.717416 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:42.717880 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:42.718213 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:36:42.718217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:36:42.718219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:36:42.718405 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:36:42.719022 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:36:42.719029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:36:42.719441 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.719764 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.720260 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":299,"threads_started":1}
I20260929 11:36:42.722975 22142 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:36:42.723280 22142 heartbeater.cc:464] Registering TS with master...
I20260929 11:36:42.724021 22142 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
I20260929 11:36:42.725497 21685 ts_manager.cc:194] Registered new tserver with Master: b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
I20260929 11:36:42.726300 21685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:56831
I20260929 11:36:42.734031 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:36:42.752036 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.738754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.6.9.254:46109)
0929 11:36:42.738987 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.738994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.739252 (+ 258us) server_negotiation.cc:407] Connection header received
0929 11:36:42.739435 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.739439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.739515 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.739599 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:42.741077 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.741818 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.743265 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.743681 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.743799 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.746648 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:42.746654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:42.746657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:42.746699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:42.749261 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.749984 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.749988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.749991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.750128 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:42.750510 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:42.750514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:42.750516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:42.750868 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:42.751139 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.751488 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.751721 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:36:42.756806 21685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:36:42.764273 21685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:36:42.793131 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.778315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37909 (local address 127.0.0.1:49094)
0929 11:36:42.781577 (+ 3262us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.781594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.781818 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.782882 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.782889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.782930 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:42.783373 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.783389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.785827 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.785833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.788898 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.788926 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.791855 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.791862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.792172 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.792251 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.792421 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.792691 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3137,"thread_start_us":2066,"threads_started":1}
I20260929 11:36:42.793131 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.778818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.6.9.195:37909)
0929 11:36:42.780209 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.780221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.781706 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 11:36:42.782021 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.782030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.782308 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.782496 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:42.783595 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.785577 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.789341 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.791207 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.791333 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.791610 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.792531 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.792772 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1047,"threads_started":1}
I20260929 11:36:42.798746 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.778309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.6.9.194:42599)
0929 11:36:42.789971 (+ 11662us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.789981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.790020 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:36:42.790158 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.790168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.790500 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.791792 (+ 1292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:42.792089 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.793764 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.795942 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.797702 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.797833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.798044 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.798286 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.798453 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11498,"thread_start_us":11376,"threads_started":1}
I20260929 11:36:42.798822 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.778161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42599 (local address 127.0.0.1:38126)
0929 11:36:42.780801 (+ 2640us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.780842 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.781025 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.791011 (+ 9986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.791017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.791059 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:42.791548 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.791563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.794075 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.794081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.795763 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.795779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.797910 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.797916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.798091 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.798126 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.798214 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.798365 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2356,"spinlock_wait_cycles":768,"thread_start_us":3142,"threads_started":1}
I20260929 11:36:42.798265 22077 tablet_service.cc:1467] Processing CreateTablet for tablet 928f0acad17e45c2b2898a7de401baef (DEFAULT_TABLE table=test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:42.799790 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.781792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:59828)
0929 11:36:42.790934 (+ 9142us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.790954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.791073 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.792065 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.792069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.792100 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:42.792494 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.792506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.794497 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.794500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.796479 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.796496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.798760 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.798769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.799120 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.799156 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.799245 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.799388 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9024,"spinlock_wait_cycles":14080,"thread_start_us":8914,"threads_started":1}
I20260929 11:36:42.800073 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928f0acad17e45c2b2898a7de401baef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:42.801400 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.782578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.6.9.193:36069)
0929 11:36:42.790449 (+ 7871us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.790457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.791210 (+ 753us) server_negotiation.cc:407] Connection header received
0929 11:36:42.791321 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.791330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.791605 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.791796 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:42.792746 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.794360 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.796749 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.798217 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.798354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.798587 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.800997 (+ 2410us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.801160 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7696,"thread_start_us":7538,"threads_started":1}
I20260929 11:36:42.803169 21943 tablet_service.cc:1467] Processing CreateTablet for tablet 928f0acad17e45c2b2898a7de401baef (DEFAULT_TABLE table=test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:42.804855 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928f0acad17e45c2b2898a7de401baef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:42.805429 21809 tablet_service.cc:1467] Processing CreateTablet for tablet 928f0acad17e45c2b2898a7de401baef (DEFAULT_TABLE table=test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:36:42.807090 21809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928f0acad17e45c2b2898a7de401baef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:36:42.816262 22160 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Bootstrap starting.
I20260929 11:36:42.820557 22160 tablet_bootstrap.cc:654] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:42.820554 22162 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Bootstrap starting.
I20260929 11:36:42.823043 22160 log.cc:824] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:42.824882 22162 tablet_bootstrap.cc:654] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:42.825762 22161 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Bootstrap starting.
I20260929 11:36:42.826368 22162 log.cc:824] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:42.829566 22161 tablet_bootstrap.cc:654] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Neither blocks nor log segments found. Creating new log.
I20260929 11:36:42.829802 22160 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: No bootstrap required, opened a new log
I20260929 11:36:42.829802 22162 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: No bootstrap required, opened a new log
I20260929 11:36:42.830123 22162 ts_tablet_manager.cc:1397] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 11:36:42.830127 22160 ts_tablet_manager.cc:1397] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260929 11:36:42.830745 22161 log.cc:824] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:36:42.834122 22161 tablet_bootstrap.cc:492] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: No bootstrap required, opened a new log
I20260929 11:36:42.834421 22161 ts_tablet_manager.cc:1397] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:36:42.841248 22162 raft_consensus.cc:348] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.841248 22160 raft_consensus.cc:348] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.841918 22160 raft_consensus.cc:374] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:42.841890 22162 raft_consensus.cc:374] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:42.842062 22160 raft_consensus.cc:729] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01ec4485fad48de94a264113b699930, State: Initialized, Role: FOLLOWER
I20260929 11:36:42.842062 22162 raft_consensus.cc:729] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc50dbec2bbb46afae886bb342cc1017, State: Initialized, Role: FOLLOWER
I20260929 11:36:42.842175 22161 raft_consensus.cc:348] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.842635 22161 raft_consensus.cc:374] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:36:42.842760 22161 raft_consensus.cc:729] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7170db9d4e8b4f0eab6d14785859b8e4, State: Initialized, Role: FOLLOWER
I20260929 11:36:42.842828 22160 consensus_queue.cc:260] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.842828 22162 consensus_queue.cc:260] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.843351 22161 consensus_queue.cc:260] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.844410 22142 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:36:42.844954 22162 ts_tablet_manager.cc:1428] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 11:36:42.844955 22160 ts_tablet_manager.cc:1428] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 11:36:42.849120 22008 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:36:42.850456 22161 ts_tablet_manager.cc:1428] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 11:36:42.910737 22168 raft_consensus.cc:482] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:36:42.911103 22168 raft_consensus.cc:504] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
W20260929 11:36:42.912160 21875 tablet.cc:2369] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:42.913376 22168 leader_election.cc:288] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b01ec4485fad48de94a264113b699930 (127.6.9.195:37909), dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069)
I20260929 11:36:42.920639 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.913979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:50519 (local address 127.6.9.195:37909)
0929 11:36:42.914208 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.914215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.914905 (+ 690us) server_negotiation.cc:407] Connection header received
0929 11:36:42.915141 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.915148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.915233 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.915309 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:42.915946 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.916798 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.918703 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.919571 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.919687 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.919800 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.920193 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.920339 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:36:42.920612 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.913839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:43537 (local address 127.6.9.193:36069)
0929 11:36:42.914050 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.914057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.915071 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 11:36:42.915225 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.915229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.915293 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.915382 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:36:42.916089 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.916849 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.918824 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.919643 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.919809 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.919911 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.920238 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.920377 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:36:42.921005 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.913850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37909 (local address 127.6.9.194:50519)
0929 11:36:42.914781 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.914801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.914961 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.915346 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.915351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.915377 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:42.915799 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.915813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.916932 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.916937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.918519 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.918540 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.919848 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.919855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.920025 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.920051 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.920131 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.920345 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":3,"thread_start_us":523,"threads_started":1}
I20260929 11:36:42.921046 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.913918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.6.9.194:43537)
0929 11:36:42.914924 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 11:36:42.914944 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:36:42.915097 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:36:42.915412 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:36:42.915417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:36:42.915448 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:36:42.915910 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.915925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.916998 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.917002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.918607 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:36:42.918625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.919900 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.919905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:36:42.920065 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.920090 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:36:42.920191 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 11:36:42.920338 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":19,"thread_start_us":603,"threads_started":1}
I20260929 11:36:42.922089 22097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928f0acad17e45c2b2898a7de401baef" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01ec4485fad48de94a264113b699930" is_pre_election: true
I20260929 11:36:42.922091 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928f0acad17e45c2b2898a7de401baef" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc50dbec2bbb46afae886bb342cc1017" is_pre_election: true
I20260929 11:36:42.922609 22097 raft_consensus.cc:2454] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 0.
I20260929 11:36:42.922613 21829 raft_consensus.cc:2454] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 0.
I20260929 11:36:42.923353 21897 leader_election.cc:302] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7170db9d4e8b4f0eab6d14785859b8e4, dc50dbec2bbb46afae886bb342cc1017; no voters:
I20260929 11:36:42.923812 22168 raft_consensus.cc:2781] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:36:42.924006 22168 raft_consensus.cc:482] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:36:42.924127 22168 raft_consensus.cc:3037] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:42.926615 22168 raft_consensus.cc:504] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
I20260929 11:36:42.927397 22168 leader_election.cc:288] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 election: Requested vote from peers b01ec4485fad48de94a264113b699930 (127.6.9.195:37909), dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069)
I20260929 11:36:42.928176 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928f0acad17e45c2b2898a7de401baef" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc50dbec2bbb46afae886bb342cc1017"
I20260929 11:36:42.928151 22097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928f0acad17e45c2b2898a7de401baef" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01ec4485fad48de94a264113b699930"
I20260929 11:36:42.928417 22097 raft_consensus.cc:3037] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:42.928417 21829 raft_consensus.cc:3037] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:36:42.931800 21829 raft_consensus.cc:2454] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 1.
I20260929 11:36:42.931800 22097 raft_consensus.cc:2454] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 1.
I20260929 11:36:42.932380 21896 leader_election.cc:302] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930; no voters:
I20260929 11:36:42.932792 22168 raft_consensus.cc:2781] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:36:42.933296 22168 raft_consensus.cc:686] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7170db9d4e8b4f0eab6d14785859b8e4, State: Running, Role: LEADER
I20260929 11:36:42.933816 22168 consensus_queue.cc:237] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } }
W20260929 11:36:42.938371 22009 tablet.cc:2369] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:42.938812 21683 catalog_manager.cc:5495] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } health_report { overall_health: UNKNOWN } } }
W20260929 11:36:42.950829 22143 tablet.cc:2369] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:36:42.987249 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:42.981909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.6.9.194:42599)
0929 11:36:42.982107 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:42.982112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:42.982150 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:36:42.982345 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:42.982350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:42.982430 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:42.982558 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:36:42.982942 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.983662 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.985190 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:42.985584 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:42.985693 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:42.986157 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:36:42.986655 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:36:42.986798 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:42.986892 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 11:36:42.986981 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:36:42.995635 21829 raft_consensus.cc:1261] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Refusing update from remote peer 7170db9d4e8b4f0eab6d14785859b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:42.995631 22097 raft_consensus.cc:1261] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Refusing update from remote peer 7170db9d4e8b4f0eab6d14785859b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:36:42.996619 22168 consensus_queue.cc:1035] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:42.996996 22171 consensus_queue.cc:1035] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:36:43.016988 22175 mvcc.cc:205] Tried to move back new op lower bound from 7334632665060048896 to 7334632664821100544. Current Snapshot: MvccSnapshot[applied={T|T < 7334632665060048896}]
I20260929 11:36:43.029091 21683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:43.029412 21683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:36:43.034018 21683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Sending DeleteTablet for 3 replicas of tablet 928f0acad17e45c2b2898a7de401baef
I20260929 11:36:43.035430 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet 928f0acad17e45c2b2898a7de401baef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:43 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:38126
I20260929 11:36:43.035512 21809 tablet_service.cc:1514] Processing DeleteTablet for tablet 928f0acad17e45c2b2898a7de401baef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:43 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59828
I20260929 11:36:43.035629 22077 tablet_service.cc:1514] Processing DeleteTablet for tablet 928f0acad17e45c2b2898a7de401baef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:36:43 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:49094
I20260929 11:36:43.036863 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21746
I20260929 11:36:43.062691 22183 tablet_replica.cc:330] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: stopping tablet replica
I20260929 11:36:43.063169 22183 raft_consensus.cc:2229] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:36:43.063803 22183 raft_consensus.cc:2258] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:43.064637 22184 tablet_replica.cc:330] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: stopping tablet replica
I20260929 11:36:43.065160 22184 raft_consensus.cc:2229] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:43.065907 22184 raft_consensus.cc:2258] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:43.066955 22185 tablet_replica.cc:330] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: stopping tablet replica
I20260929 11:36:43.067430 22185 raft_consensus.cc:2229] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:36:43.067987 22185 raft_consensus.cc:2258] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:36:43.070212 22185 ts_tablet_manager.cc:1905] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:43.073760 22184 ts_tablet_manager.cc:1905] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:36:43.073760 22183 ts_tablet_manager.cc:1905] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:36:43 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:36:43.077582 22185 ts_tablet_manager.cc:1918] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:43.077785 22185 log.cc:1196] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/928f0acad17e45c2b2898a7de401baef
I20260929 11:36:43.078321 22185 ts_tablet_manager.cc:1939] T 928f0acad17e45c2b2898a7de401baef P b01ec4485fad48de94a264113b699930: Deleting consensus metadata
I20260929 11:36:43.080173 21669 catalog_manager.cc:4841] TS b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): tablet 928f0acad17e45c2b2898a7de401baef (table test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]) successfully deleted
I20260929 11:36:43.081472 22184 ts_tablet_manager.cc:1918] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:43.081676 22184 log.cc:1196] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/928f0acad17e45c2b2898a7de401baef
I20260929 11:36:43.082237 22184 ts_tablet_manager.cc:1939] T 928f0acad17e45c2b2898a7de401baef P dc50dbec2bbb46afae886bb342cc1017: Deleting consensus metadata
I20260929 11:36:43.084316 21670 catalog_manager.cc:4841] TS dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069): tablet 928f0acad17e45c2b2898a7de401baef (table test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]) successfully deleted
I20260929 11:36:43.086051 22183 ts_tablet_manager.cc:1918] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:36:43.086266 22183 log.cc:1196] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/928f0acad17e45c2b2898a7de401baef
I20260929 11:36:43.086355 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:43.038055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.6.9.193:36069)
0929 11:36:43.038263 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:43.038270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:43.060804 (+ 22534us) server_negotiation.cc:407] Connection header received
0929 11:36:43.061408 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:43.061415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:43.061532 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:43.061629 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:43.064218 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.065621 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.070620 (+ 4999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.071279 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.071443 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:43.077655 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:43.077676 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:43.077698 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:43.077756 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:43.082098 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.083286 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.083298 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.083301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.083490 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.084007 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.084012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.084016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.084490 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:43.084822 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:43.085461 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 11:36:43.085720 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:36:43.086835 22183 ts_tablet_manager.cc:1939] T 928f0acad17e45c2b2898a7de401baef P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting consensus metadata
I20260929 11:36:43.087984 21869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:43.088646 21670 catalog_manager.cc:4841] TS 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): tablet 928f0acad17e45c2b2898a7de401baef (table test-table [id=dc9bc83f466f4e9f8cb780391e5ee384]) successfully deleted
I20260929 11:36:43.182840 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21746
I20260929 11:36:43.206873 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21880
Sep 29 11:36:43 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:36:43.224161 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:43.207755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.6.9.194:42599)
0929 11:36:43.207928 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:43.207933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:43.207953 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:36:43.208090 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:43.208096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:43.208190 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:43.208302 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:43.209871 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.210736 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.212230 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.212772 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.212924 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:43.217546 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:43.217565 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:43.217582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:43.217633 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:43.220903 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.221701 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.221710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.221712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.221982 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.222447 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.222451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.222453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.222869 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:43.223244 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:43.223560 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 11:36:43.223801 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:36:43.224927 22003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:43.341440 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21880
I20260929 11:36:43.357982 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22014
Sep 29 11:36:43 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:36:43.375551 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:36:43.358911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.6.9.195:37909)
0929 11:36:43.359167 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 11:36:43.359174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:36:43.359266 (+ 92us) server_negotiation.cc:407] Connection header received
0929 11:36:43.359462 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:36:43.359468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:36:43.359544 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:36:43.359651 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:36:43.361067 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.361924 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.363620 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:36:43.364057 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:36:43.364181 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:36:43.369200 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:36:43.369221 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:36:43.369238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:36:43.369281 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:36:43.372281 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.373319 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.373328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.373330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.373503 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:36:43.373929 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:36:43.373933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:36:43.373935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:36:43.374338 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:36:43.374648 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:36:43.374962 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 11:36:43.375193 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 11:36:43.376253 22137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:43.470147 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22014
I20260929 11:36:43.483974 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 21653
I20260929 11:36:43.485280 21715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:36:43.596652 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 21653
I20260929 11:37:03.615069 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--webserver_interface=127.6.9.254
--webserver_port=39665
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:03.954411 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:03.954732 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:03.954872 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:03.962558 22191 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:37:03.962683 22191 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:37:03.962724 22191 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:37:03.962793 22191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:03.962868 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:03.962961 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:03.963016 22191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:03.963091 22191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:03.973898 22191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:03.975311 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:03.976701 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:04.000491 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.000459 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.000592 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:04.000805 22191 server_base.cc:1034] running on GCE node
I20260929 11:37:04.002177 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:04.003983 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:04.005280 22191 hybrid_clock.cc:648] HybridClock initialized: now 1790681824005197 us; error 62 us; skew 500 ppm
I20260929 11:37:04.005718 22191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:04 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:04.009657 22191 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:37:04.011495 22191 webserver.cc:460] Webserver started at http://127.6.9.254:39665/ using document root <none> and password file <none>
I20260929 11:37:04.012383 22191 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:04.012528 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:04.019577 22191 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:37:04.023535 22207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:04.025321 22191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:37:04.025544 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c8cded25f3314a47b7b5a05e5530c464"
format_stamp: "Formatted at 2026-09-29 11:36:40 on dist-test-slave-c0l9"
server_key: "b9063dab6e2a7d46e496ae067ae8bcec"
server_key_iv: "d35ff53d302d176271a3a5240bf40289"
server_key_version: "encryptionkey@0"
I20260929 11:37:04.026743 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:04.063968 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:04.068828 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:04.069135 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:04.088359 22191 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46109
I20260929 11:37:04.088404 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46109 every 8 connection(s)
I20260929 11:37:04.089978 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:04.097399 22259 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:37:04.099787 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22191
I20260929 11:37:04.100696 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:36069
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=39295
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:37:04.101212 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap starting.
I20260929 11:37:04.122562 22259 log.cc:824] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:04.137441 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:37:04.138085 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap complete.
I20260929 11:37:04.147836 22259 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:04.148854 22259 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Initialized, Role: FOLLOWER
I20260929 11:37:04.149572 22259 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:04.149824 22259 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:04.149955 22259 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:04.150133 22259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:37:04.154418 22259 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:04.155023 22259 leader_election.cc:302] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8cded25f3314a47b7b5a05e5530c464; no voters:
I20260929 11:37:04.155719 22259 leader_election.cc:288] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:37:04.155882 22263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:37:04.157202 22263 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 LEADER]: Becoming Leader. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Running, Role: LEADER
I20260929 11:37:04.157835 22263 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:04.158332 22259 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:04.160991 22265 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8cded25f3314a47b7b5a05e5530c464. Latest consensus state: current_term: 2 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:37:04.161083 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:37:04.161588 22265 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:04.161648 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:04.174216 22270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:04.183368 22270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:04.185073 22270 catalog_manager.cc:1259] Loaded cluster ID: eea8e8f5b07b4ccca60678980746a7d8
I20260929 11:37:04.185286 22270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:04.189438 22270 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:04.191219 22270 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Loaded TSK: 0
I20260929 11:37:04.225018 22270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Generated new TSK 1
I20260929 11:37:04.225493 22270 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:37:04.473357 22261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:04.473701 22261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:04.473850 22261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:04.481858 22261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:04.482016 22261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:04.482188 22261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:04.492926 22261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=39295
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:04.494374 22261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:04.495723 22261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:04.510489 22287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.510515 22286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.511587 22289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:04.512408 22261 server_base.cc:1034] running on GCE node
I20260929 11:37:04.513201 22261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:04.514290 22261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:04.515524 22261 hybrid_clock.cc:648] HybridClock initialized: now 1790681824515470 us; error 41 us; skew 500 ppm
I20260929 11:37:04.515846 22261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:04 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:04.519307 22261 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:37:04.521092 22261 webserver.cc:460] Webserver started at http://127.6.9.193:39295/ using document root <none> and password file <none>
I20260929 11:37:04.521991 22261 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:04.522140 22261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:04.529347 22261 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 11:37:04.533141 22296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:04.534886 22261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 11:37:04.535096 22261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "dc50dbec2bbb46afae886bb342cc1017"
format_stamp: "Formatted at 2026-09-29 11:36:41 on dist-test-slave-c0l9"
server_key: "e76826240805893b5473ab7299188609"
server_key_iv: "45d638d02d0f0002a5dbbf4c382890d4"
server_key_version: "encryptionkey@0"
I20260929 11:37:04.536546 22261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:04.557263 22261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:04.563131 22261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:04.563439 22261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:04.564648 22261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:04.566447 22261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:04.566562 22261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:04.566684 22261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:04.566756 22261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:04.598580 22261 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:36069
I20260929 11:37:04.598645 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:36069 every 8 connection(s)
I20260929 11:37:04.600056 22261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:04.605280 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22261
I20260929 11:37:04.605898 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:42599
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=37525
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:37:04 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:04.627318 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:04.603024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44087 (local address 127.6.9.254:46109)
0929 11:37:04.603670 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:04.603683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:04.604446 (+ 763us) server_negotiation.cc:407] Connection header received
0929 11:37:04.606864 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:04.606885 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:04.607403 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:04.607821 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:04.609384 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:04.611309 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:04.613272 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:04.614017 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:04.614180 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:04.618552 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:04.618565 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:04.618583 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:04.618634 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:04.622044 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:04.623185 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:04.623195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:04.623201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:04.623416 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:04.624044 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:04.624050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:04.624053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:04.624769 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:04.624952 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:04.625984 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0929 11:37:04.626338 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":289,"threads_started":1}
I20260929 11:37:04.627667 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:04.603300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:44087)
0929 11:37:04.604204 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:04.604259 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:04.606206 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:04.607921 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:04.607928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:04.608481 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:04.609111 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:04.609132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:04.611526 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:04.611531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:04.613082 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:04.613101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:04.614287 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:04.614295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:04.614584 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:04.615333 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:04.615363 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:04.618125 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:04.622273 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:04.622285 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:04.622305 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:04.622996 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:04.623575 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:04.623582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:04.623586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:04.623874 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:04.624916 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:04.624926 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:04.625390 (+ 464us) client_negotiation.cc:769] Sending connection context
0929 11:37:04.625742 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 11:37:04.626368 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":362,"threads_started":1}
I20260929 11:37:04.630697 22409 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:37:04.631019 22409 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:04.631839 22409 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
I20260929 11:37:04.634261 22224 ts_manager.cc:194] Registered new tserver with Master: dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069)
I20260929 11:37:04.635865 22224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:44087
W20260929 11:37:04.946710 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:04.947057 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:04.947207 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:04.954622 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:04.954770 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:04.954926 22413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:04.965432 22413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=37525
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:04.966907 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:04.968293 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:04.982034 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.982033 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:04.982055 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:04.983310 22413 server_base.cc:1034] running on GCE node
I20260929 11:37:04.983964 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:04.985054 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:04.986263 22413 hybrid_clock.cc:648] HybridClock initialized: now 1790681824986223 us; error 33 us; skew 500 ppm
I20260929 11:37:04.986593 22413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:04 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:04.989822 22413 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:37:04.991356 22413 webserver.cc:460] Webserver started at http://127.6.9.194:37525/ using document root <none> and password file <none>
I20260929 11:37:04.992059 22413 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:04.992189 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:04.998039 22413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:37:05.001438 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.003247 22413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:37:05.003473 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "7170db9d4e8b4f0eab6d14785859b8e4"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "4001d537f2a0d2a563c93013d9aa6c1d"
server_key_iv: "f7847a739b54c6bfc55651cebf5ccb81"
server_key_version: "encryptionkey@0"
I20260929 11:37:05.004750 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:05.028999 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:05.034044 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:05.034312 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:05.035364 22413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:05.036873 22413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:05.036983 22413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.037091 22413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:05.037159 22413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.066495 22413 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42599
I20260929 11:37:05.066565 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42599 every 8 connection(s)
I20260929 11:37:05.067947 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:05.072846 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22413
I20260929 11:37:05.073470 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:37909
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=36397
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:37:05 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:05.091256 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.070503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34663 (local address 127.6.9.254:46109)
0929 11:37:05.070735 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.070740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.071924 (+ 1184us) server_negotiation.cc:407] Connection header received
0929 11:37:05.074064 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.074070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.074188 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.074281 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:05.076316 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.077496 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.079720 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.080323 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.080461 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.084901 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:05.084909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:05.084912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:05.084953 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:05.087534 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.088561 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.088568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.088572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.088748 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.089244 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.089248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.089251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.089647 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:05.089805 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.090664 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.090933 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:37:05.092546 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.070954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:34663)
0929 11:37:05.071723 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.071778 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.073767 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.074564 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.074573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.075168 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:05.076078 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.076107 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.077765 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.077770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.079542 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.079556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.080557 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.080562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.080846 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.081705 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:05.081745 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:05.084564 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:05.087741 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:05.087751 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:05.087771 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:05.088400 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:05.088857 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:05.088861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:05.088864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:05.089128 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:05.089782 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:05.089790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:05.090226 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.090553 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.091133 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
I20260929 11:37:05.094189 22541 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:37:05.094542 22541 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:05.095500 22541 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
I20260929 11:37:05.097371 22224 ts_manager.cc:194] Registered new tserver with Master: 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599)
I20260929 11:37:05.098497 22224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:34663
W20260929 11:37:05.415959 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:05.416301 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:05.416429 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:05.424345 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:05.424518 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:05.424722 22544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:05.435084 22544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:37909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=36397
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:05.436537 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:05.438324 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:05.453202 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:05.453239 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:05.453276 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:05.454005 22544 server_base.cc:1034] running on GCE node
I20260929 11:37:05.454703 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:05.455782 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:05.456990 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790681825456939 us; error 55 us; skew 500 ppm
I20260929 11:37:05.457317 22544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:05 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:05.460740 22544 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:37:05.462335 22544 webserver.cc:460] Webserver started at http://127.6.9.195:36397/ using document root <none> and password file <none>
I20260929 11:37:05.463059 22544 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:05.463200 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:05.469049 22544 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:37:05.471917 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.473392 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:37:05.473574 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b01ec4485fad48de94a264113b699930"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "3a7e58606bc2a2f3ee4196374493f35f"
server_key_iv: "b91c631d5dcbcd056db1e09e5bdf4338"
server_key_version: "encryptionkey@0"
I20260929 11:37:05.474630 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:05.492564 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:05.497309 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:05.497591 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:05.498737 22544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:05.500260 22544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:05.500355 22544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.500469 22544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:05.500557 22544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:05.533910 22544 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:37909
I20260929 11:37:05.533985 22671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:37909 every 8 connection(s)
I20260929 11:37:05.535923 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:05.541195 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22544
Sep 29 11:37:05 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681825, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:05.559041 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.542251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.6.9.254:46109)
0929 11:37:05.545022 (+ 2771us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.545026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.545041 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:37:05.545084 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.545087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.545144 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.545230 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:05.547236 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.548030 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.549442 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.549859 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.549968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.553082 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:05.553089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:05.553092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:05.553137 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:05.556011 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.556758 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.556765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.556768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.556917 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.557332 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.557335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.557337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.557637 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:05.557911 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.558208 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.558505 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2677,"thread_start_us":2558,"threads_started":1}
I20260929 11:37:05.559304 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.539273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:59213 (local address 127.6.9.254:46109)
0929 11:37:05.539489 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.539495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.540956 (+ 1461us) server_negotiation.cc:407] Connection header received
0929 11:37:05.543326 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.543334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.543437 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.543555 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:05.545475 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.546219 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.548160 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.548616 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.548754 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.552739 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:05.552747 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:05.552752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:05.552794 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:05.555973 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.556845 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.556851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.556855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.556984 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.557452 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.557457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.557459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.557733 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:05.557865 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.558669 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.558911 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:37:05.560473 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.539846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:59213)
0929 11:37:05.540767 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.540816 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.542936 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.543798 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.543808 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.544538 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:05.545252 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.545278 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.546390 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.546396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.548000 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.548015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.548884 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.548891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.549136 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.549913 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:05.549944 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:05.552462 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:05.556185 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:05.556194 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:05.556211 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:05.556725 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:05.557111 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:05.557114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:05.557116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:05.557326 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:05.557866 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:05.557875 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:05.558252 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.558513 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.559111 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":366,"threads_started":1}
I20260929 11:37:05.562208 22672 heartbeater.cc:347] Connected to a master server at 127.6.9.254:46109
I20260929 11:37:05.562556 22672 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:05.563310 22672 heartbeater.cc:511] Master 127.6.9.254:46109 requested a full tablet report, sending...
I20260929 11:37:05.564772 22224 ts_manager.cc:194] Registered new tserver with Master: b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
I20260929 11:37:05.565569 22224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:59213
I20260929 11:37:05.575253 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:37:05.583009 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.576267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.6.9.254:46109)
0929 11:37:05.576474 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.576479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.576926 (+ 447us) server_negotiation.cc:407] Connection header received
0929 11:37:05.577113 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.577118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.577281 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.577525 (+ 244us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:05.578135 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.578987 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.580977 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.581425 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.581543 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.582094 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:05.582339 (+ 245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 11:37:05.582746 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34300: token expired
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 11:37:05.583154 22411 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34300: token expired
W20260929 11:37:05.583851 22147 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.9.254:46109)
I20260929 11:37:05.599032 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.584483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.6.9.254:46109)
0929 11:37:05.584683 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.584689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.584707 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:05.584879 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.584883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.584948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.585023 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:05.586922 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.587732 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.589847 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.590431 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.590567 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.593407 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:05.593415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:05.593419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:05.593463 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:05.596117 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.596851 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.596858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.596860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.596992 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:05.597462 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:05.597467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:05.597469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:05.597844 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:05.598113 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.598484 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.598718 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 11:37:05.602736 22224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:37:05.610172 22224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:37:05.639385 22409 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:37:05.644034 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.624144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.6.9.193:36069)
0929 11:37:05.634975 (+ 10831us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.634985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.635019 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:37:05.635269 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.635284 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.635589 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.635938 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:05.636745 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.638146 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.641121 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.642649 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.642779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.643032 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.643514 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.643702 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10622,"thread_start_us":10739,"threads_started":1}
I20260929 11:37:05.644106 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.623711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39554)
0929 11:37:05.634033 (+ 10322us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.634073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.634200 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.635964 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.635973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.636007 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:05.636438 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.636455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.638317 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.638323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.640878 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.640893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.642988 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.643000 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.643241 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.643308 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.643454 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.643685 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10173,"thread_start_us":10576,"threads_started":1}
I20260929 11:37:05.645673 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.623425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.6.9.195:37909)
0929 11:37:05.636859 (+ 13434us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.636868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.636904 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:37:05.637003 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.637011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.637314 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.637551 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:05.638249 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.640175 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.643116 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.644509 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.644629 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.644882 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.645195 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.645366 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13203,"thread_start_us":13053,"threads_started":1}
I20260929 11:37:05.645709 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.623026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37909 (local address 127.0.0.1:41554)
0929 11:37:05.632431 (+ 9405us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.632464 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.632622 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.637582 (+ 4960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.637587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.637623 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:05.638035 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.638051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.640346 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.640351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.642878 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.642894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.644760 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.644767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.644959 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.645009 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.645113 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.645258 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9236,"mutex_wait_us":772,"thread_start_us":8503,"threads_started":1}
I20260929 11:37:05.646230 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.622518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42599 (local address 127.0.0.1:51444)
0929 11:37:05.623290 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.623327 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.623761 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.638333 (+ 14572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.638339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.638369 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:05.638878 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.638895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.640881 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.640888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.643664 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.643679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.645530 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.645536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.645733 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.645768 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.645842 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.645953 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":571,"thread_start_us":386,"threads_started":1}
I20260929 11:37:05.647159 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.622693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.6.9.194:42599)
0929 11:37:05.637710 (+ 15017us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.637721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.637751 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:37:05.637854 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.637862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.638120 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.638336 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:05.639117 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.640656 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.643875 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.645265 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.646429 (+ 1164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.646609 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.646668 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.646826 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14837,"thread_start_us":15881,"threads_started":1}
I20260929 11:37:05.648826 22344 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 (DEFAULT_TABLE table=test-table [id=16524083fa4d4944bc208bffd9668edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:05.650501 22344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:05.651389 22607 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 (DEFAULT_TABLE table=test-table [id=16524083fa4d4944bc208bffd9668edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:05.651517 22476 tablet_service.cc:1467] Processing CreateTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 (DEFAULT_TABLE table=test-table [id=16524083fa4d4944bc208bffd9668edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:05.652808 22607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:05.652951 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:05.667265 22688 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Bootstrap starting.
I20260929 11:37:05.670783 22688 tablet_bootstrap.cc:654] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:05.671839 22689 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Bootstrap starting.
I20260929 11:37:05.672294 22688 log.cc:824] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:05.673480 22690 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Bootstrap starting.
I20260929 11:37:05.677434 22689 tablet_bootstrap.cc:654] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:05.679183 22689 log.cc:824] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:05.679437 22690 tablet_bootstrap.cc:654] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:05.680847 22690 log.cc:824] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:05.682212 22688 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: No bootstrap required, opened a new log
I20260929 11:37:05.682488 22689 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: No bootstrap required, opened a new log
I20260929 11:37:05.682564 22688 ts_tablet_manager.cc:1397] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260929 11:37:05.682786 22689 ts_tablet_manager.cc:1397] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 11:37:05.683584 22690 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: No bootstrap required, opened a new log
I20260929 11:37:05.683852 22690 ts_tablet_manager.cc:1397] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 11:37:05.691602 22690 raft_consensus.cc:348] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.692039 22690 raft_consensus.cc:374] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:05.692167 22690 raft_consensus.cc:729] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7170db9d4e8b4f0eab6d14785859b8e4, State: Initialized, Role: FOLLOWER
I20260929 11:37:05.692853 22690 consensus_queue.cc:260] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.693789 22688 raft_consensus.cc:348] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.693789 22689 raft_consensus.cc:348] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.694139 22541 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:37:05.694254 22689 raft_consensus.cc:374] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:05.694329 22688 raft_consensus.cc:374] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:05.694401 22689 raft_consensus.cc:729] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc50dbec2bbb46afae886bb342cc1017, State: Initialized, Role: FOLLOWER
I20260929 11:37:05.694461 22688 raft_consensus.cc:729] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01ec4485fad48de94a264113b699930, State: Initialized, Role: FOLLOWER
I20260929 11:37:05.694870 22690 ts_tablet_manager.cc:1428] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 11:37:05.695178 22688 consensus_queue.cc:260] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.695169 22689 consensus_queue.cc:260] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.696756 22672 heartbeater.cc:503] Master 127.6.9.254:46109 was elected leader, sending a full tablet report...
I20260929 11:37:05.696983 22689 ts_tablet_manager.cc:1428] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 11:37:05.697392 22688 ts_tablet_manager.cc:1428] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260929 11:37:05.788172 22673 tablet.cc:2369] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:37:05.819947 22542 tablet.cc:2369] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:37:05.852519 22410 tablet.cc:2369] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:05.889766 22694 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:05.890105 22694 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.892007 22694 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
I20260929 11:37:05.899940 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.892541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:34437 (local address 127.6.9.193:36069)
0929 11:37:05.892866 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.892874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.893909 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 11:37:05.894120 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.894125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.894201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.894298 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:05.895034 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.895749 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.898050 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.898899 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.899026 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.899138 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.899528 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.899687 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 11:37:05.899968 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.892697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.6.9.194:34437)
0929 11:37:05.893784 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.893809 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.893959 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.894340 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.894345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.894389 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:05.894863 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.894878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.895880 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.895884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.897885 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.897901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.899125 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.899131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.899314 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.899350 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.899453 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.899580 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":36,"thread_start_us":781,"threads_started":1}
I20260929 11:37:05.900710 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.892886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:39323 (local address 127.6.9.195:37909)
0929 11:37:05.893482 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:05.893491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:05.893741 (+ 250us) server_negotiation.cc:407] Connection header received
0929 11:37:05.893825 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:05.893830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:05.893907 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:05.893988 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:05.894858 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.895617 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.898260 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.899463 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.899564 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.899672 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:05.900277 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 11:37:05.900430 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260929 11:37:05.901463 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:05.892371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37909 (local address 127.6.9.194:39323)
0929 11:37:05.893427 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:05.893448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:05.893653 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:05.894176 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:05.894180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:05.894207 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:05.894695 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.894710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.895758 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.895763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.898050 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:05.898066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:05.899873 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:05.899879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:05.900083 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:05.900115 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:37:05.900900 (+ 785us) client_negotiation.cc:240] Negotiation successful
0929 11:37:05.901069 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":44,"thread_start_us":686,"threads_started":1}
I20260929 11:37:05.901285 22364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe00a94dfb4ed99a0538a47f35c6d9" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc50dbec2bbb46afae886bb342cc1017" is_pre_election: true
I20260929 11:37:05.901906 22364 raft_consensus.cc:2454] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 0.
I20260929 11:37:05.902758 22430 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7170db9d4e8b4f0eab6d14785859b8e4, dc50dbec2bbb46afae886bb342cc1017; no voters:
I20260929 11:37:05.903236 22694 raft_consensus.cc:2781] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:37:05.903366 22694 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:37:05.903465 22694 raft_consensus.cc:3037] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:05.903685 22627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe00a94dfb4ed99a0538a47f35c6d9" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01ec4485fad48de94a264113b699930" is_pre_election: true
I20260929 11:37:05.904301 22627 raft_consensus.cc:2454] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 0.
I20260929 11:37:05.906061 22694 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.906740 22694 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 election: Requested vote from peers dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
I20260929 11:37:05.907328 22627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe00a94dfb4ed99a0538a47f35c6d9" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01ec4485fad48de94a264113b699930"
I20260929 11:37:05.907303 22364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bbe00a94dfb4ed99a0538a47f35c6d9" candidate_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc50dbec2bbb46afae886bb342cc1017"
I20260929 11:37:05.907546 22627 raft_consensus.cc:3037] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:05.907531 22364 raft_consensus.cc:3037] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:05.910600 22627 raft_consensus.cc:2454] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 1.
I20260929 11:37:05.910876 22364 raft_consensus.cc:2454] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7170db9d4e8b4f0eab6d14785859b8e4 in term 1.
I20260929 11:37:05.911158 22429 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930; no voters:
I20260929 11:37:05.911612 22694 raft_consensus.cc:2781] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:05.912081 22694 raft_consensus.cc:686] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7170db9d4e8b4f0eab6d14785859b8e4, State: Running, Role: LEADER
I20260929 11:37:05.912606 22694 consensus_queue.cc:237] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:05.917891 22224 catalog_manager.cc:5495] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } health_report { overall_health: UNKNOWN } } }
I20260929 11:37:06.005311 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.000037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.6.9.194:42599)
0929 11:37:06.000253 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:06.000258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:06.000276 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:06.000363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:06.000367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:06.000452 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:06.000573 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:06.000911 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.001613 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.002963 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.003391 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.003496 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:06.004076 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:06.004610 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:37:06.004834 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:06.004949 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 11:37:06.005037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:37:06.013875 22364 raft_consensus.cc:1261] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Refusing update from remote peer 7170db9d4e8b4f0eab6d14785859b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:06.013876 22627 raft_consensus.cc:1261] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Refusing update from remote peer 7170db9d4e8b4f0eab6d14785859b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:06.014935 22699 consensus_queue.cc:1035] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:06.015307 22694 consensus_queue.cc:1035] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:06.029307 22702 mvcc.cc:205] Tried to move back new op lower bound from 7334632759342522368 to 7334632758942408704. Current Snapshot: MvccSnapshot[applied={T|T < 7334632759342522368}]
I20260929 11:37:06.043475 22224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:06.043797 22224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:06.047904 22224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Sending DeleteTablet for 3 replicas of tablet 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:06.049338 22607 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:41554
I20260929 11:37:06.049504 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22261
I20260929 11:37:06.049394 22344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:39554
I20260929 11:37:06.049639 22476 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:06 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:51444
I20260929 11:37:06.066470 22712 tablet_replica.cc:330] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: stopping tablet replica
I20260929 11:37:06.067112 22712 raft_consensus.cc:2229] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:06.072432 22713 tablet_replica.cc:330] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: stopping tablet replica
I20260929 11:37:06.071679 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.051127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.6.9.193:36069)
0929 11:37:06.051329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:06.051336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:06.051356 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:37:06.051418 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:06.051424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:06.051509 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:06.051611 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:06.053783 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.055097 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.057337 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.058038 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.058174 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:06.062336 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:06.062364 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:06.062388 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:06.062448 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:06.066955 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.067989 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.067998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.068001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.068198 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.068723 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.068728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.068732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.069268 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:06.069606 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:06.070973 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0929 11:37:06.071253 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:37:06.073231 22712 raft_consensus.cc:2258] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:06.073701 22404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:06.074216 22713 raft_consensus.cc:2229] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:37:06.074631 22711 tablet_replica.cc:330] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: stopping tablet replica
I20260929 11:37:06.074908 22713 raft_consensus.cc:2258] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:06.075112 22711 raft_consensus.cc:2229] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:06.076525 22711 raft_consensus.cc:2258] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:06.077606 22713 ts_tablet_manager.cc:1905] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:06.078950 22711 ts_tablet_manager.cc:1905] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:06.085948 22713 ts_tablet_manager.cc:1918] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:06.085942 22711 ts_tablet_manager.cc:1918] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:06.086146 22711 log.cc:1196] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:06.086352 22713 log.cc:1196] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:06.086740 22711 ts_tablet_manager.cc:1939] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P b01ec4485fad48de94a264113b699930: Deleting consensus metadata
I20260929 11:37:06.087056 22713 ts_tablet_manager.cc:1939] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P 7170db9d4e8b4f0eab6d14785859b8e4: Deleting consensus metadata
I20260929 11:37:06.088907 22209 catalog_manager.cc:4841] TS 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 (table test-table [id=16524083fa4d4944bc208bffd9668edf]) successfully deleted
I20260929 11:37:06.088907 22208 catalog_manager.cc:4841] TS b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): tablet 4bbe00a94dfb4ed99a0538a47f35c6d9 (table test-table [id=16524083fa4d4944bc208bffd9668edf]) successfully deleted
I20260929 11:37:06.186163 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22261
I20260929 11:37:06.201969 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22413
W20260929 11:37:06.202015 22209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.9.193:36069 (error 108)
I20260929 11:37:06.206314 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.203506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39576)
0929 11:37:06.205933 (+ 2427us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.206083 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2289,"thread_start_us":2185,"threads_started":1}
W20260929 11:37:06.206676 22209 catalog_manager.cc:4584] TS dc50dbec2bbb46afae886bb342cc1017 (127.6.9.193:36069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bbe00a94dfb4ed99a0538a47f35c6d9: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
I20260929 11:37:06.221764 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.203222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.6.9.194:42599)
0929 11:37:06.203435 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:06.203441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:06.203461 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:37:06.203526 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:06.203531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:06.203612 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:06.203768 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:06.205630 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.206589 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.208022 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.208535 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.208958 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:06.215199 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:06.215217 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:06.215234 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:06.215293 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:06.218474 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.219362 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.219378 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.219380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.219558 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.220029 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.220035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.220039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.220486 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:06.220962 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:06.221135 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 11:37:06.221374 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:37:06.222513 22536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:06.260603 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.260125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39578)
0929 11:37:06.260350 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.260456 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:37:06.261874 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.261429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39586)
0929 11:37:06.261642 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.261722 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 11:37:06.322377 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.321897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39592)
0929 11:37:06.322114 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.322223 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 11:37:06.323422 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.323047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39598)
0929 11:37:06.323200 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.323275 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:37:06.336855 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22413
I20260929 11:37:06.352409 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22544
I20260929 11:37:06.371834 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.353309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.6.9.195:37909)
0929 11:37:06.353505 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:06.353509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:06.353548 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:37:06.353730 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:06.353735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:06.353814 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:06.353905 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:06.355528 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.356281 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.357859 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:06.358391 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:06.358559 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:06.365868 (+ 7309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:06.365888 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:06.365905 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:06.365969 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:06.368897 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.369781 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.369790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.369792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.369951 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:06.370400 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:06.370404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:06.370406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:06.370791 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:06.371017 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:06.371325 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 11:37:06.371509 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:37:06.372577 22667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:06.404832 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.404235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39602)
0929 11:37:06.404503 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.404603 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 11:37:06.406032 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.405546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39616)
0929 11:37:06.405781 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.405883 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 11:37:06.463581 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22544
I20260929 11:37:06.478055 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22191
I20260929 11:37:06.479210 22254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:06.596448 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:06.595885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:36069 (local address 127.0.0.1:39624)
0929 11:37:06.596158 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:06.596267 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 11:37:06.596524 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22191
I20260929 11:37:26.614497 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--webserver_interface=127.6.9.254
--webserver_port=39665
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:26.953258 22720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:26.953595 22720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:26.953739 22720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:26.961905 22720 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:37:26.962049 22720 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:37:26.962105 22720 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:37:26.962188 22720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:26.962261 22720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:26.962361 22720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:26.962424 22720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:26.962499 22720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:26.972833 22720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:26.974193 22720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:26.975574 22720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:26.987319 22725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:26.987306 22726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:26.987833 22728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:26.988255 22720 server_base.cc:1034] running on GCE node
I20260929 11:37:26.989097 22720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:26.990609 22720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:26.991865 22720 hybrid_clock.cc:648] HybridClock initialized: now 1790681846991796 us; error 34 us; skew 500 ppm
I20260929 11:37:26.992283 22720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:26 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681846, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:27.001595 22720 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:37:27.003283 22720 webserver.cc:460] Webserver started at http://127.6.9.254:39665/ using document root <none> and password file <none>
I20260929 11:37:27.004033 22720 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:27.004137 22720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:27.009832 22720 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:37:27.013779 22735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:27.015164 22720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:37:27.015340 22720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c8cded25f3314a47b7b5a05e5530c464"
format_stamp: "Formatted at 2026-09-29 11:36:40 on dist-test-slave-c0l9"
server_key: "b9063dab6e2a7d46e496ae067ae8bcec"
server_key_iv: "d35ff53d302d176271a3a5240bf40289"
server_key_version: "encryptionkey@0"
I20260929 11:37:27.016351 22720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:27.033035 22720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:27.037606 22720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:27.037894 22720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:27.056376 22720 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46109
I20260929 11:37:27.056409 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46109 every 8 connection(s)
I20260929 11:37:27.057955 22720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:27.058822 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22720
I20260929 11:37:27.059681 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:36069
--local_ip_for_outbound_sockets=127.6.9.193
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=39295
--webserver_interface=127.6.9.193
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:37:27.065220 22787 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:37:27.068452 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap starting.
I20260929 11:37:27.089182 22787 log.cc:824] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:27.106297 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:37:27.106873 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Bootstrap complete.
I20260929 11:37:27.115999 22787 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:27.117012 22787 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Initialized, Role: FOLLOWER
I20260929 11:37:27.117719 22787 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:27.117980 22787 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:27.118095 22787 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:27.118247 22787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:37:27.122685 22787 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:27.123215 22787 leader_election.cc:302] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8cded25f3314a47b7b5a05e5530c464; no voters:
I20260929 11:37:27.123853 22787 leader_election.cc:288] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:37:27.124006 22791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:37:27.126106 22787 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:27.131011 22791 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [term 3 LEADER]: Becoming Leader. State: Replica: c8cded25f3314a47b7b5a05e5530c464, State: Running, Role: LEADER
I20260929 11:37:27.131783 22791 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } }
I20260929 11:37:27.135691 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:37:27.136150 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:27.136783 22793 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8cded25f3314a47b7b5a05e5530c464. Latest consensus state: current_term: 3 leader_uuid: "c8cded25f3314a47b7b5a05e5530c464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cded25f3314a47b7b5a05e5530c464" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46109 } } }
I20260929 11:37:27.137118 22793 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:27.139375 22802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:27.148619 22802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:27.150381 22802 catalog_manager.cc:1259] Loaded cluster ID: eea8e8f5b07b4ccca60678980746a7d8
I20260929 11:37:27.150576 22802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:27.155231 22802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:27.158284 22802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Loaded TSK: 0
I20260929 11:37:27.158692 22802 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8cded25f3314a47b7b5a05e5530c464: Loaded TSK: 1
F20260929 11:37:27.160109 22802 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38c49669ad google::LogMessage::Fail() at ??:0
@ 0x7f38c496988a google::LogMessage::SendToLog() at ??:0
@ 0x7f38c49664de google::LogMessage::Flush() at ??:0
@ 0x7f38c4967a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38c86ca7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f38c86b973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f38da3951c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f38da394783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f38da4202a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f38da4200c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f38da46a5e9 std::function<>::operator()() at ??:0
@ 0x7f38da398cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f38da396c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38da41e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f38da41e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f38c75b2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f38c75b591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f38c75b5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f38c7590eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f38ca86a6db start_thread at ??:0
@ 0x7f38c380061f clone at ??:0
W20260929 11:37:27.474524 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:27.474807 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:27.474902 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:27.482468 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:27.482600 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:27.482741 22789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:27.492686 22789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=39295
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:27.493985 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:27.495253 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:27.508129 22835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:27.508114 22836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:27.508338 22838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:27.508989 22789 server_base.cc:1034] running on GCE node
I20260929 11:37:27.509650 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:27.511358 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:27.519016 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790681847518294 us; error 670 us; skew 500 ppm
I20260929 11:37:27.519507 22789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:27 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681847, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:27.523276 22789 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:37:27.525041 22789 webserver.cc:460] Webserver started at http://127.6.9.193:39295/ using document root <none> and password file <none>
I20260929 11:37:27.525800 22789 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:27.525916 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:27.536708 22789 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:37:27.540072 22845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:27.541703 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:37:27.541926 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "dc50dbec2bbb46afae886bb342cc1017"
format_stamp: "Formatted at 2026-09-29 11:36:41 on dist-test-slave-c0l9"
server_key: "e76826240805893b5473ab7299188609"
server_key_iv: "45d638d02d0f0002a5dbbf4c382890d4"
server_key_version: "encryptionkey@0"
I20260929 11:37:27.542953 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:27.563728 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:27.568419 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:27.568796 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:27.570024 22789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:27.571995 22852 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 11:37:27.578614 22789 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 11:37:27.578827 22789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 11:37:27.578965 22789 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 11:37:27.582572 22789 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 11:37:27.582670 22789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260929 11:37:27.583067 22852 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Bootstrap starting.
I20260929 11:37:27.592190 22852 log.cc:824] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:27.610281 22852 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:37:27.611393 22852 tablet_bootstrap.cc:492] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Bootstrap complete.
I20260929 11:37:27.612032 22852 ts_tablet_manager.cc:1397] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.008s
I20260929 11:37:27.623531 22959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:36069 every 8 connection(s)
I20260929 11:37:27.623526 22789 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:36069
I20260929 11:37:27.625047 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:27.624010 22852 raft_consensus.cc:348] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:27.625200 22852 raft_consensus.cc:729] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc50dbec2bbb46afae886bb342cc1017, State: Initialized, Role: FOLLOWER
I20260929 11:37:27.626008 22852 consensus_queue.cc:260] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:27.627825 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22789
I20260929 11:37:27.628619 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:42599
--local_ip_for_outbound_sockets=127.6.9.194
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=37525
--webserver_interface=127.6.9.194
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:37:27.639818 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:27.628159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:50303)
0929 11:37:27.637879 (+ 9720us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:27.638223 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9238,"threads_started":1}
W20260929 11:37:27.641937 22960 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46109 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:46109: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
I20260929 11:37:27.643611 22852 ts_tablet_manager.cc:1428] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017: Time spent starting tablet: real 0.031s user 0.021s sys 0.008s
I20260929 11:37:27.643635 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:27.643253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:47519)
0929 11:37:27.643432 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:27.643507 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:37:27.644634 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:27.644285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:33103)
0929 11:37:27.644439 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:27.644502 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:37:27.644850 22960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:37:27.966410 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:27.966713 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:27.966840 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:27.974048 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:27.974208 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:27.974368 22964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:27.984256 22964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:42599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=37525
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:27.985561 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:27.986878 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:27.999481 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:27.999507 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:27.999437 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:28.000550 22964 server_base.cc:1034] running on GCE node
I20260929 11:37:28.001236 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:28.002177 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:28.003377 22964 hybrid_clock.cc:648] HybridClock initialized: now 1790681848003314 us; error 56 us; skew 500 ppm
I20260929 11:37:28.003679 22964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:28 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681848, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:28.006578 22964 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:37:28.007999 22964 webserver.cc:460] Webserver started at http://127.6.9.194:37525/ using document root <none> and password file <none>
I20260929 11:37:28.008692 22964 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:28.008809 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:28.014211 22964 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 11:37:28.017026 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.018483 22964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:37:28.018643 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "7170db9d4e8b4f0eab6d14785859b8e4"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "4001d537f2a0d2a563c93013d9aa6c1d"
server_key_iv: "f7847a739b54c6bfc55651cebf5ccb81"
server_key_version: "encryptionkey@0"
I20260929 11:37:28.019659 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:28.053242 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:28.057341 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:28.057602 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:28.058622 22964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:28.059940 22964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:28.060040 22964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.060154 22964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:28.060225 22964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.090107 22964 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42599
I20260929 11:37:28.090167 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42599 every 8 connection(s)
I20260929 11:37:28.091569 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:28.096421 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.094617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:50727)
0929 11:37:28.095345 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.095549 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":287,"threads_started":1}
W20260929 11:37:28.097028 23092 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46109 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:46109: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
I20260929 11:37:28.097882 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.097537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:50507)
0929 11:37:28.097692 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.097761 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:37:28.098744 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.098388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:44397)
0929 11:37:28.098536 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.098623 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:37:28.098976 23092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:37:28.099566 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 22964
I20260929 11:37:28.100154 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:37909
--local_ip_for_outbound_sockets=127.6.9.195
--tserver_master_addrs=127.6.9.254:46109
--webserver_port=36397
--webserver_interface=127.6.9.195
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:28.420450 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:28.420794 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:28.420917 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:28.428404 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:28.428531 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:28.428709 23095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:28.438591 23095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:37909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=36397
--tserver_master_addrs=127.6.9.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:28.439924 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:28.441305 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:28.454505 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:28.454583 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:28.454555 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:28.455467 23095 server_base.cc:1034] running on GCE node
I20260929 11:37:28.456130 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:28.457175 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:28.458402 23095 hybrid_clock.cc:648] HybridClock initialized: now 1790681848458333 us; error 57 us; skew 500 ppm
I20260929 11:37:28.458739 23095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:28 dist-test-slave-c0l9 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681848, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:28.462232 23095 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:37:28.463825 23095 webserver.cc:460] Webserver started at http://127.6.9.195:36397/ using document root <none> and password file <none>
I20260929 11:37:28.464537 23095 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:28.464716 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:28.470306 23095 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:37:28.473016 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.474300 23095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:37:28.474455 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "b01ec4485fad48de94a264113b699930"
format_stamp: "Formatted at 2026-09-29 11:36:42 on dist-test-slave-c0l9"
server_key: "3a7e58606bc2a2f3ee4196374493f35f"
server_key_iv: "b91c631d5dcbcd056db1e09e5bdf4338"
server_key_version: "encryptionkey@0"
I20260929 11:37:28.475435 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:28.489171 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:28.493458 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:28.493727 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:28.494747 23095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:28.496114 23095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:28.496222 23095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.496332 23095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:28.496402 23095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:28.526510 23095 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:37909
I20260929 11:37:28.526564 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:37909 every 8 connection(s)
I20260929 11:37:28.527897 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:28.532004 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23095
I20260929 11:37:28.533056 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.530859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:42309)
0929 11:37:28.531612 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.531885 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":275,"threads_started":1}
W20260929 11:37:28.533892 23223 heartbeater.cc:644] Failed to heartbeat to 127.6.9.254:46109 (0 consecutive failures): Network error: Failed to ping master at 127.6.9.254:46109: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
I20260929 11:37:28.534749 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.534383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:59651)
0929 11:37:28.534547 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.534635 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 11:37:28.535661 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.535254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:49361)
0929 11:37:28.535443 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.535540 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 11:37:28.535892 23223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:37:28.646413 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.645484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:37595)
0929 11:37:28.646048 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.646132 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
I20260929 11:37:28.871583 23228 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:28.871884 23228 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:28.873512 23228 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
Sep 29 11:37:28 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681847, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
Sep 29 11:37:28 dist-test-slave-c0l9 krb5kdc[21640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681847, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:37:28.894745 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.874130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:50565 (local address 127.6.9.194:42599)
0929 11:37:28.876262 (+ 2132us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:28.876276 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:28.876322 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:37:28.878412 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:28.878424 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:28.878858 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:28.879190 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:28.880864 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.882035 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.883946 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.884550 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.884738 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:28.888250 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:28.888261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:28.888279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:28.888323 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:28.891129 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:28.891971 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:28.891977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:28.891983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:28.892143 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:28.892583 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:28.892586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:28.892588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:28.893153 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:28.893345 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:28.894141 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 11:37:28.894453 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1958,"thread_start_us":1843,"threads_started":1}
I20260929 11:37:28.894785 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.873931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:37909 (local address 127.6.9.193:38019)
0929 11:37:28.874081 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.874118 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:28.874795 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:28.878990 (+ 4195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:28.879002 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:28.879917 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:28.880649 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:28.880699 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.882363 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.882368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:28.883810 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:28.883822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.884783 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.884790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:28.885063 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:28.885791 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:28.885809 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:28.888341 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:28.891404 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:28.891407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:28.891410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:28.891839 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:28.892252 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:28.892257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:28.892261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:28.892471 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:28.893465 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:28.893470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:28.893639 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 11:37:28.893919 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 11:37:28.894394 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":61,"spinlock_wait_cycles":13952}
I20260929 11:37:28.895076 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.874158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:38019 (local address 127.6.9.195:37909)
0929 11:37:28.875691 (+ 1533us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:28.875706 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:28.875751 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:37:28.877926 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:28.877939 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:28.878480 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:28.878891 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:28.880871 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.882189 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.883939 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.884546 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.884743 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:28.888564 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:28.888571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:28.888581 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:28.888616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:28.891279 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:28.891970 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:28.891977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:28.891983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:28.892138 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:28.892589 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:28.892593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:28.892595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:28.893227 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:28.893398 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:28.894329 (+ 931us) server_negotiation.cc:299] Negotiation successful
0929 11:37:28.894705 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1117,"threads_started":1}
I20260929 11:37:28.895617 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:28.873931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42599 (local address 127.6.9.193:50565)
0929 11:37:28.875978 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:28.875997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:28.876116 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:28.879151 (+ 3035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:28.879155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:28.879890 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:28.880650 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:28.880700 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.882227 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.882233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:28.883815 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:28.883827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:28.884804 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:28.884811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:28.885061 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:28.885728 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:28.885759 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:28.887955 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:28.891290 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:28.891299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:28.891313 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:28.891848 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:28.892258 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:28.892263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:28.892267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:28.892471 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:28.893297 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:28.893305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:28.893593 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 11:37:28.894938 (+ 1345us) client_negotiation.cc:240] Negotiation successful
0929 11:37:28.895221 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"mutex_wait_us":359,"spinlock_wait_cycles":23424,"thread_start_us":1503,"threads_started":1}
W20260929 11:37:28.898512 22847 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
W20260929 11:37:28.898818 22846 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:28.898939 22846 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc50dbec2bbb46afae886bb342cc1017; no voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930
I20260929 11:37:28.899310 23228 raft_consensus.cc:2726] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:37:29.100555 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:29.099586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:41785)
0929 11:37:29.100132 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:29.100274 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":238,"threads_started":1}
I20260929 11:37:29.537408 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:29.536554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:36273)
0929 11:37:29.537074 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:29.537193 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260929 11:37:29.648212 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:29.647398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:33317)
0929 11:37:29.647891 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:29.648005 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260929 11:37:30.102310 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:30.101579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:40599)
0929 11:37:30.102063 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:30.102145 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":273,"threads_started":1}
I20260929 11:37:30.539178 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:30.538396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:59399)
0929 11:37:30.538898 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:30.538987 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
I20260929 11:37:30.649922 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:30.649145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:60119)
0929 11:37:30.649655 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:30.649743 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260929 11:37:30.709079 23238 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:30.709365 23238 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:30.710167 23238 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
W20260929 11:37:30.711167 22847 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
W20260929 11:37:30.711371 22846 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:30.711504 22846 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc50dbec2bbb46afae886bb342cc1017; no voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930
I20260929 11:37:30.711766 23238 raft_consensus.cc:2726] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:37:31.104110 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:31.103380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:59085)
0929 11:37:31.103855 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:31.103942 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260929 11:37:31.540778 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:31.540009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:46029)
0929 11:37:31.540477 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:31.540572 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":249,"threads_started":1}
I20260929 11:37:31.651492 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:31.650798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:40467)
0929 11:37:31.651218 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:31.651313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260929 11:37:32.105839 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:32.105105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:33759)
0929 11:37:32.105570 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:32.105656 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260929 11:37:32.542692 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:32.541878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:59363)
0929 11:37:32.542377 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:32.542478 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":234,"threads_started":1}
I20260929 11:37:32.653355 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:32.652556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:37281)
0929 11:37:32.653072 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:32.653172 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
I20260929 11:37:33.107726 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:33.106923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:46887)
0929 11:37:33.107465 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:33.107555 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
I20260929 11:37:33.544371 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:33.543641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:42141)
0929 11:37:33.544096 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:33.544204 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260929 11:37:33.654948 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:33.654169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:55921)
0929 11:37:33.654690 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:33.654781 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":325,"threads_started":1}
I20260929 11:37:33.729027 23248 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:33.729220 23248 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:33.729836 23248 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
W20260929 11:37:33.730858 22847 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
W20260929 11:37:33.731050 22846 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:33.731150 22846 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc50dbec2bbb46afae886bb342cc1017; no voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930
I20260929 11:37:33.731361 23248 raft_consensus.cc:2726] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:37:34.109637 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:34.108835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:40899)
0929 11:37:34.109360 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:34.109450 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
I20260929 11:37:34.546276 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:34.545411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:54505)
0929 11:37:34.545964 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:34.546050 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":301,"threads_started":1}
I20260929 11:37:34.656816 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:34.655958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:58367)
0929 11:37:34.656477 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:34.656583 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":270,"threads_started":1}
I20260929 11:37:35.111639 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:35.110715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:58413)
0929 11:37:35.111319 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:35.111422 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":286,"threads_started":1}
I20260929 11:37:35.548251 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:35.547312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:49561)
0929 11:37:35.547872 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:35.547968 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260929 11:37:35.658736 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:35.657884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:46171)
0929 11:37:35.658405 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:35.658515 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260929 11:37:36.113653 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:36.112803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:44893)
0929 11:37:36.113302 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:36.113422 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260929 11:37:36.549964 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:36.549260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:39877)
0929 11:37:36.549698 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:36.549782 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260929 11:37:36.660476 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:36.659707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:41633)
0929 11:37:36.660204 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:36.660288 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":241,"threads_started":1}
I20260929 11:37:37.115564 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:37.114691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:45473)
0929 11:37:37.115299 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:37.115396 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":297,"threads_started":1}
I20260929 11:37:37.551859 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:37.551027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:35181)
0929 11:37:37.551603 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:37.551695 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
I20260929 11:37:37.662313 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:37.661555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:42263)
0929 11:37:37.662066 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:37.662154 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260929 11:37:37.762493 23261 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:37.762728 23261 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:37.763453 23261 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
W20260929 11:37:37.764523 22846 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
W20260929 11:37:37.764751 22847 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:37.764914 22847 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc50dbec2bbb46afae886bb342cc1017; no voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930
I20260929 11:37:37.765221 23261 raft_consensus.cc:2726] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:37:38.117406 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:38.116597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:34783)
0929 11:37:38.117135 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:38.117215 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
I20260929 11:37:38.553802 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:38.553026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:47313)
0929 11:37:38.553505 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:38.553613 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260929 11:37:38.664182 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:38.663345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:47341)
0929 11:37:38.663873 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:38.663966 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":296,"threads_started":1}
I20260929 11:37:39.119078 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:39.118270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:55245)
0929 11:37:39.118785 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:39.118884 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
I20260929 11:37:39.555776 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:39.554862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:37187)
0929 11:37:39.555519 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:39.555612 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":395,"threads_started":1}
I20260929 11:37:39.666101 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:39.665345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:43867)
0929 11:37:39.665821 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:39.665918 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":257,"threads_started":1}
I20260929 11:37:40.120908 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:40.120095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:43485)
0929 11:37:40.120590 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:40.120716 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260929 11:37:40.542546 23269 raft_consensus.cc:482] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:40.542853 23269 raft_consensus.cc:504] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7170db9d4e8b4f0eab6d14785859b8e4" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42599 } } peers { permanent_uuid: "dc50dbec2bbb46afae886bb342cc1017" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 36069 } } peers { permanent_uuid: "b01ec4485fad48de94a264113b699930" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 37909 } }
I20260929 11:37:40.543509 23269 leader_election.cc:288] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599), b01ec4485fad48de94a264113b699930 (127.6.9.195:37909)
W20260929 11:37:40.544525 22846 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b01ec4485fad48de94a264113b699930 (127.6.9.195:37909): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
W20260929 11:37:40.544711 22847 leader_election.cc:341] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7170db9d4e8b4f0eab6d14785859b8e4 (127.6.9.194:42599): Not found: Tablet not found: 4bbe00a94dfb4ed99a0538a47f35c6d9
I20260929 11:37:40.544853 22847 leader_election.cc:302] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc50dbec2bbb46afae886bb342cc1017; no voters: 7170db9d4e8b4f0eab6d14785859b8e4, b01ec4485fad48de94a264113b699930
I20260929 11:37:40.545122 23269 raft_consensus.cc:2726] T 4bbe00a94dfb4ed99a0538a47f35c6d9 P dc50dbec2bbb46afae886bb342cc1017 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:37:40.557883 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:40.557029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:48257)
0929 11:37:40.557601 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:40.557687 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":291,"threads_started":1}
I20260929 11:37:40.668134 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:40.667254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:51357)
0929 11:37:40.667775 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:40.667889 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":272,"threads_started":1}
I20260929 11:37:41.122808 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:41.121990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:48989)
0929 11:37:41.122532 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:41.122627 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":265,"threads_started":1}
I20260929 11:37:41.559798 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:41.558990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:40787)
0929 11:37:41.559513 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:41.559610 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260929 11:37:41.670027 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:41.669181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:58727)
0929 11:37:41.669725 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:41.669824 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
I20260929 11:37:42.124831 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:42.123882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:59965)
0929 11:37:42.124509 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:42.124597 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":327,"threads_started":1}
I20260929 11:37:42.561755 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:42.560932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:55679)
0929 11:37:42.561439 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:42.561530 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
I20260929 11:37:42.672122 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:42.671287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.193:59463)
0929 11:37:42.671792 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:42.671893 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":246,"threads_started":1}
I20260929 11:37:43.126751 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:43.125972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.194:55623)
0929 11:37:43.126484 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:43.126580 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260929 11:37:43.538471 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22789
I20260929 11:37:43.554688 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:43.539853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.6.9.193:36069)
0929 11:37:43.540438 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:43.540448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:43.540477 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:37:43.540546 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:43.540552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:43.540839 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:43.540976 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:43.542465 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.543441 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.544942 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.545552 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.545658 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:43.548530 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:43.548544 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:43.548563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:43.548632 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:43.551743 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.552583 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.552591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.552593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.552817 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.553251 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.553255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.553257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.553683 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:43.553933 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:43.554196 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:37:43.554420 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":317,"threads_started":1}
I20260929 11:37:43.556048 22955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:43.563642 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:43.562880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:46109 (local address 127.6.9.195:41401)
0929 11:37:43.563394 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:43.563488 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.254:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
I20260929 11:37:43.649300 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22789
I20260929 11:37:43.664765 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 22964
I20260929 11:37:43.677886 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:43.665527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.6.9.194:42599)
0929 11:37:43.666008 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:43.666013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:43.666029 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:37:43.666089 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:43.666093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:43.666154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:43.666248 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:43.667668 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.668487 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.669898 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.670327 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.670440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:43.672993 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:43.672998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:43.673001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:43.673037 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:43.675463 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.676083 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.676087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.676089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.676227 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.676604 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.676607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.676609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.676919 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:43.677138 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:43.677422 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 11:37:43.677599 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":264,"threads_started":1}
I20260929 11:37:43.678633 23087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:43.765134 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 22964
I20260929 11:37:43.779695 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23095
I20260929 11:37:43.794939 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:43.780844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.6.9.195:37909)
0929 11:37:43.781399 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:43.781403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:43.781421 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:43.781474 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:43.781478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:43.781547 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:43.781670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:43.783069 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.783875 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.785486 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:43.785925 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:43.786048 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:43.788770 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:43.788776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:43.788780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:43.788822 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:43.791769 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.792566 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.792571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.792573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.792917 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:43.793396 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:43.793402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:43.793405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:43.793702 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:43.794007 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:43.794299 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 11:37:43.794550 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260929 11:37:43.795812 23218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:43.886070 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23095
2026-09-29T11:37:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64601 ms)
[----------] 30 tests from SecurityITest (297322 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:37:43 dist-test-slave-c0l9 krb5kdc[23288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:37:43 dist-test-slave-c0l9 krb5kdc[23288](info): set up 2 sockets
Sep 29 11:37:43 dist-test-slave-c0l9 krb5kdc[23288](info): commencing operation
krb5kdc: starting...
W20260929 11:37:44.924651 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:37:45 dist-test-slave-c0l9 krb5kdc[23288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:37:45.173024 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44159
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:45.529865 23301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:45.530225 23301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:45.530367 23301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:45.537948 23301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:45.538087 23301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:45.538170 23301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:45.538249 23301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:45.538309 23301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:45.548381 23301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:44159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:45.549762 23301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:45.551148 23301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:45.562191 23306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:45.563024 23307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:45.563094 23309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:45.564067 23301 server_base.cc:1034] running on GCE node
I20260929 11:37:45.564903 23301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:45.566308 23301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:45.567512 23301 hybrid_clock.cc:648] HybridClock initialized: now 1790681865567466 us; error 34 us; skew 500 ppm
I20260929 11:37:45.567837 23301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:45 dist-test-slave-c0l9 krb5kdc[23288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:45.570948 23301 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:37:45.572571 23301 webserver.cc:460] Webserver started at http://127.6.9.254:39661/ using document root <none> and password file <none>
I20260929 11:37:45.573340 23301 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:45.573451 23301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:45.573734 23301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:45.576921 23301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "1af8fccdd2e74b03ac7fa7e690d2032f"
format_stamp: "Formatted at 2026-09-29 11:37:45 on dist-test-slave-c0l9"
server_key: "b6e886f4452cb933655c338fe813a9fd"
server_key_iv: "50fd0c425eb0fadf4b9845fbf2f6b9bd"
server_key_version: "encryptionkey@0"
I20260929 11:37:45.577670 23301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "1af8fccdd2e74b03ac7fa7e690d2032f"
format_stamp: "Formatted at 2026-09-29 11:37:45 on dist-test-slave-c0l9"
server_key: "b6e886f4452cb933655c338fe813a9fd"
server_key_iv: "50fd0c425eb0fadf4b9845fbf2f6b9bd"
server_key_version: "encryptionkey@0"
I20260929 11:37:45.582873 23301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:37:45.586387 23316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:45.587772 23301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:37:45.587945 23301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "1af8fccdd2e74b03ac7fa7e690d2032f"
format_stamp: "Formatted at 2026-09-29 11:37:45 on dist-test-slave-c0l9"
server_key: "b6e886f4452cb933655c338fe813a9fd"
server_key_iv: "50fd0c425eb0fadf4b9845fbf2f6b9bd"
server_key_version: "encryptionkey@0"
I20260929 11:37:45.588136 23301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:45.603299 23301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:45.607946 23301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:45.608229 23301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:45.626477 23301 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:44159
I20260929 11:37:45.626506 23367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:44159 every 8 connection(s)
I20260929 11:37:45.627940 23301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:45.631834 23368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:45.638222 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23301
I20260929 11:37:45.638423 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:37:45.639061 6183 external_mini_cluster.cc:1428] Setting key 9cc2acde6f0693194f7619a5c23983d7
I20260929 11:37:45.643867 23368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: Bootstrap starting.
I20260929 11:37:45.647806 23368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:45.649133 23368 log.cc:824] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:45.652405 23368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: No bootstrap required, opened a new log
Sep 29 11:37:45 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:45.662194 23368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } }
I20260929 11:37:45.662719 23368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:45.662835 23368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af8fccdd2e74b03ac7fa7e690d2032f, State: Initialized, Role: FOLLOWER
I20260929 11:37:45.663550 23368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } }
I20260929 11:37:45.663849 23368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:45.663992 23368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:45.664170 23368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:45.665232 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:45.641159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.6.9.254:44159)
0929 11:37:45.641990 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:45.642006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:45.642055 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:37:45.644431 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:45.644467 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:45.645007 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:45.645445 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:45.646784 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:45.648767 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:45.650488 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:45.651506 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:45.651699 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:45.655953 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:45.655964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:45.655991 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:45.656039 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:45.660129 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:45.660966 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:45.660981 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:45.660990 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:45.661220 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:45.661635 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:45.661640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:45.661643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:45.662397 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:45.662686 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:45.663338 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 11:37:45.663800 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":348,"threads_started":1}
I20260929 11:37:45.666932 23368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } }
I20260929 11:37:45.667428 23368 leader_election.cc:302] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af8fccdd2e74b03ac7fa7e690d2032f; no voters:
I20260929 11:37:45.667999 23368 leader_election.cc:288] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:37:45.668138 23373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:45.668886 23373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [term 1 LEADER]: Becoming Leader. State: Replica: 1af8fccdd2e74b03ac7fa7e690d2032f, State: Running, Role: LEADER
I20260929 11:37:45.669384 23373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } }
I20260929 11:37:45.670122 23368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:45.671906 23374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } } }
I20260929 11:37:45.671877 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af8fccdd2e74b03ac7fa7e690d2032f. Latest consensus state: current_term: 1 leader_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af8fccdd2e74b03ac7fa7e690d2032f" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44159 } } }
I20260929 11:37:45.672302 23374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:45.672302 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:45.673987 23378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:45.679523 23378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:45.688743 23378 catalog_manager.cc:1347] Generated new cluster ID: 63606c686ee8432fbfad3dd3e0cad608
I20260929 11:37:45.688931 23378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:45.700857 23378 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:37:45.702718 23378 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:45.716001 23378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: Generated new TSK 0
I20260929 11:37:45.716995 23378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:37:45.780742 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:46.141696 23394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:46.142015 23394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:46.142112 23394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:46.149547 23394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:46.149689 23394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:46.149811 23394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:46.159981 23394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:46.161381 23394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:46.162707 23394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:46.176340 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:46.176271 23402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:46.176221 23399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:46.177134 23394 server_base.cc:1034] running on GCE node
I20260929 11:37:46.177785 23394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:46.178839 23394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:46.180050 23394 hybrid_clock.cc:648] HybridClock initialized: now 1790681866179992 us; error 50 us; skew 500 ppm
I20260929 11:37:46.180375 23394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:46 dist-test-slave-c0l9 krb5kdc[23288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:46.184011 23394 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:37:46.185667 23394 webserver.cc:460] Webserver started at http://127.6.9.193:35897/ using document root <none> and password file <none>
I20260929 11:37:46.186383 23394 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:46.186504 23394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:46.186796 23394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:46.190043 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "0a36b6d6d5ce4e9b89673a079492c0a5"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "aaee89e417c9763a6a8c5647be4fb919"
server_key_iv: "950ebf3affca15137ca47df95d297a5b"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.191109 23394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "0a36b6d6d5ce4e9b89673a079492c0a5"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "aaee89e417c9763a6a8c5647be4fb919"
server_key_iv: "950ebf3affca15137ca47df95d297a5b"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.197361 23394 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:37:46.201395 23409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.202893 23394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:46.203068 23394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "0a36b6d6d5ce4e9b89673a079492c0a5"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "aaee89e417c9763a6a8c5647be4fb919"
server_key_iv: "950ebf3affca15137ca47df95d297a5b"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.203291 23394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:46.220000 23394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:46.225594 23394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:46.225893 23394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:46.226882 23394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:46.228273 23394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:46.228354 23394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.228489 23394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:46.228547 23394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.257905 23394 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:34143
I20260929 11:37:46.257968 23521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:34143 every 8 connection(s)
I20260929 11:37:46.259218 23394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:46.262487 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23394
I20260929 11:37:46.262694 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:37:46.263526 6183 external_mini_cluster.cc:1428] Setting key 80c4a3ce3de35c1040a67c6d94659333
Sep 29 11:37:46 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:46.282728 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:46.262023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:55457 (local address 127.6.9.254:44159)
0929 11:37:46.262550 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:46.262556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:46.263801 (+ 1245us) server_negotiation.cc:407] Connection header received
0929 11:37:46.266266 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:46.266271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:46.266349 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:46.266448 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:46.268802 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.269585 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.271865 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.272285 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.272400 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:46.276856 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:46.276862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:46.276866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:46.276908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:46.279449 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:46.280271 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:46.280276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:46.280279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:46.280417 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:46.280919 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:46.280923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:46.280925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:46.281199 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:46.281344 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:46.282115 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 11:37:46.282367 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260929 11:37:46.283743 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:46.262562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44159 (local address 127.6.9.193:55457)
0929 11:37:46.263508 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:46.263565 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:46.265896 (+ 2331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:46.266794 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:46.266810 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:46.267635 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:46.268533 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:46.268562 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.269767 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.269772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:46.271679 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:46.271697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.272545 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.272552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:46.272880 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:46.273651 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:46.273682 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:46.276429 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:46.279610 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:46.279620 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:46.279644 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:46.280140 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:46.280527 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:46.280532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:46.280536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:46.280810 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:46.281326 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:46.281335 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:46.281722 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 11:37:46.281979 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 11:37:46.282542 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":338,"threads_started":1}
I20260929 11:37:46.285437 23522 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44159
I20260929 11:37:46.285743 23522 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:46.286513 23522 heartbeater.cc:511] Master 127.6.9.254:44159 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
I20260929 11:37:46.289040 23333 ts_manager.cc:194] Registered new tserver with Master: 0a36b6d6d5ce4e9b89673a079492c0a5 (127.6.9.193:34143)
I20260929 11:37:46.290570 23333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:55457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:37:46.319681 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:46.655356 23528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:46.655652 23528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:46.655772 23528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:46.663110 23528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:46.663230 23528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:46.663388 23528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:46.673743 23528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:46.675127 23528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:46.676613 23528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:46.689911 23533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:46.689935 23534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:46.690399 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:46.690713 23528 server_base.cc:1034] running on GCE node
I20260929 11:37:46.691395 23528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:46.692452 23528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:46.693660 23528 hybrid_clock.cc:648] HybridClock initialized: now 1790681866693620 us; error 36 us; skew 500 ppm
I20260929 11:37:46.693996 23528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:46 dist-test-slave-c0l9 krb5kdc[23288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:46.701458 23528 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:37:46.703190 23528 webserver.cc:460] Webserver started at http://127.6.9.194:36365/ using document root <none> and password file <none>
I20260929 11:37:46.703938 23528 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:46.704079 23528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:46.704401 23528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:46.707695 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "bb0d448258ad40b180bc9f0db0f2b10d"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "385db1691da64bedffcd906220cc792d"
server_key_iv: "4ecc3c8bac1cdd7ec3eb302bf1ddafd1"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.708560 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0d448258ad40b180bc9f0db0f2b10d"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "385db1691da64bedffcd906220cc792d"
server_key_iv: "4ecc3c8bac1cdd7ec3eb302bf1ddafd1"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.714203 23528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:37:46.718010 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.719561 23528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260929 11:37:46.719774 23528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "bb0d448258ad40b180bc9f0db0f2b10d"
format_stamp: "Formatted at 2026-09-29 11:37:46 on dist-test-slave-c0l9"
server_key: "385db1691da64bedffcd906220cc792d"
server_key_iv: "4ecc3c8bac1cdd7ec3eb302bf1ddafd1"
server_key_version: "encryptionkey@0"
I20260929 11:37:46.720010 23528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:46.737571 23528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:46.741923 23528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:46.742199 23528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:46.743186 23528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:46.744529 23528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:46.744604 23528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.744750 23528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:46.744802 23528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:46.776197 23528 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:34609
I20260929 11:37:46.776226 23655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:34609 every 8 connection(s)
I20260929 11:37:46.777691 23528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:46.783981 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23528
I20260929 11:37:46.784142 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:37:46.784582 6183 external_mini_cluster.cc:1428] Setting key 12779b43378c61c7d5e7ba480ae65307
Sep 29 11:37:46 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681866, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:46.799091 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:46.780255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:56495 (local address 127.6.9.254:44159)
0929 11:37:46.780417 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:46.780421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:46.781640 (+ 1219us) server_negotiation.cc:407] Connection header received
0929 11:37:46.783496 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:46.783502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:46.783594 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:46.783682 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:46.785444 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.786390 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.788107 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.788776 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.788902 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:46.792647 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:46.792653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:46.792656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:46.792714 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:46.795679 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:46.796617 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:46.796621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:46.796624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:46.796810 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:46.797309 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:46.797315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:46.797317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:46.797645 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:46.797766 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:46.798633 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 11:37:46.798831 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":49}
I20260929 11:37:46.800473 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:46.780740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44159 (local address 127.6.9.194:56495)
0929 11:37:46.781460 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:46.781498 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:46.783213 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:46.783931 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:46.783940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:46.784491 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:46.785253 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:46.785281 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.786560 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.786565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:46.787934 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:46.787950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:46.789002 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:46.789007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:46.789235 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:46.789900 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:46.789930 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:46.792375 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:46.795889 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:46.795899 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:46.795918 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:46.796475 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:46.796919 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:46.796924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:46.796928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:46.797173 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:46.797784 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:46.797801 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:46.798248 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 11:37:46.798545 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 11:37:46.799105 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":287,"threads_started":1}
I20260929 11:37:46.802291 23656 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44159
I20260929 11:37:46.802625 23656 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:46.803385 23656 heartbeater.cc:511] Master 127.6.9.254:44159 requested a full tablet report, sending...
I20260929 11:37:46.804939 23333 ts_manager.cc:194] Registered new tserver with Master: bb0d448258ad40b180bc9f0db0f2b10d (127.6.9.194:34609)
I20260929 11:37:46.805766 23333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:56495
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:37:46.840647 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:47.180552 23661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:47.180953 23661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:47.181113 23661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:47.188766 23661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:47.188897 23661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:47.189069 23661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:47.199312 23661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:47.200644 23661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:47.201960 23661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:47.215057 23666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:47.215011 23669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:47.214982 23667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:47.215732 23661 server_base.cc:1034] running on GCE node
I20260929 11:37:47.216358 23661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:47.217507 23661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:47.218729 23661 hybrid_clock.cc:648] HybridClock initialized: now 1790681867218671 us; error 39 us; skew 500 ppm
I20260929 11:37:47.219081 23661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:47 dist-test-slave-c0l9 krb5kdc[23288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:47.222229 23661 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:37:47.223760 23661 webserver.cc:460] Webserver started at http://127.6.9.195:42027/ using document root <none> and password file <none>
I20260929 11:37:47.224552 23661 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:47.224725 23661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:47.225157 23661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:47.229002 23661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "67deab5803d24d4cafcf59bca13e7a1c"
format_stamp: "Formatted at 2026-09-29 11:37:47 on dist-test-slave-c0l9"
server_key: "10c1d213b6d14ccc85659a0b2cc57002"
server_key_iv: "9e44cff509487cf0ee7a285a5af55181"
server_key_version: "encryptionkey@0"
I20260929 11:37:47.229836 23661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "67deab5803d24d4cafcf59bca13e7a1c"
format_stamp: "Formatted at 2026-09-29 11:37:47 on dist-test-slave-c0l9"
server_key: "10c1d213b6d14ccc85659a0b2cc57002"
server_key_iv: "9e44cff509487cf0ee7a285a5af55181"
server_key_version: "encryptionkey@0"
I20260929 11:37:47.235283 23661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:37:47.238824 23676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:47.240223 23661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:37:47.240392 23661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "67deab5803d24d4cafcf59bca13e7a1c"
format_stamp: "Formatted at 2026-09-29 11:37:47 on dist-test-slave-c0l9"
server_key: "10c1d213b6d14ccc85659a0b2cc57002"
server_key_iv: "9e44cff509487cf0ee7a285a5af55181"
server_key_version: "encryptionkey@0"
I20260929 11:37:47.240587 23661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:47.284056 23661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:47.288517 23661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:47.288856 23661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:47.289844 23661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:47.291265 23661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:47.291363 23661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:47.291486 23661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:47.291551 23661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:47.294968 23522 heartbeater.cc:503] Master 127.6.9.254:44159 was elected leader, sending a full tablet report...
I20260929 11:37:47.322413 23661 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:46463
I20260929 11:37:47.322484 23788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:46463 every 8 connection(s)
I20260929 11:37:47.323640 23661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:47.324862 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23661
I20260929 11:37:47.325037 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:37:47.325567 6183 external_mini_cluster.cc:1428] Setting key 3aebf8399cfb66e6af4fb02106ef5a28
Sep 29 11:37:47 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681867, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:47.344074 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.326468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54099 (local address 127.6.9.254:44159)
0929 11:37:47.327099 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.327105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.327924 (+ 819us) server_negotiation.cc:407] Connection header received
0929 11:37:47.329941 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.329947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.330031 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.330127 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:47.331641 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.332431 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.334047 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.334464 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.334576 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.338089 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:47.338094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:47.338097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:47.338140 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:47.341158 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.342000 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.342005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.342007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.342135 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.342535 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.342539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.342541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.342808 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:47.342941 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.343573 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.343749 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":354,"threads_started":1}
I20260929 11:37:47.345191 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.327050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44159 (local address 127.6.9.195:54099)
0929 11:37:47.327781 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.327826 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.329678 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.330381 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.330390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.330900 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:47.331477 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.331495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.332584 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.332588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.333901 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.333915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.334689 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.334696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.334933 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.335547 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:47.335570 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:47.337863 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:47.341307 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:47.341317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:47.341337 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:47.341885 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:47.342217 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:47.342220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:47.342222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:47.342424 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:47.342924 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:47.342931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:47.343266 (+ 335us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.343501 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.343949 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":273,"threads_started":1}
I20260929 11:37:47.346637 23789 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44159
I20260929 11:37:47.346930 23789 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:47.347852 23789 heartbeater.cc:511] Master 127.6.9.254:44159 requested a full tablet report, sending...
I20260929 11:37:47.349488 23333 ts_manager.cc:194] Registered new tserver with Master: 67deab5803d24d4cafcf59bca13e7a1c (127.6.9.195:46463)
I20260929 11:37:47.350307 23333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:54099
I20260929 11:37:47.354233 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:37:47.371093 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.358612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.6.9.254:44159)
0929 11:37:47.358795 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.358801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.359256 (+ 455us) server_negotiation.cc:407] Connection header received
0929 11:37:47.359386 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.359390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.359455 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.359534 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:47.361082 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.361904 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.363114 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.363515 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.363612 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.365983 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:47.365988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:47.365991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:47.366024 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:47.368420 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.369131 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.369136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.369138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.369290 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.369606 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.369609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.369611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.369957 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:47.370152 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.370517 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.370707 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:37:47.375137 23333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:37:47.382081 23333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:37:47.419199 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.395712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.6.9.195:46463)
0929 11:37:47.408223 (+ 12511us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.408234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.410094 (+ 1860us) server_negotiation.cc:407] Connection header received
0929 11:37:47.410262 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.410270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.410545 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.410762 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:47.412178 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.413988 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.416582 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.417979 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.418095 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.418310 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.418632 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.418811 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12261,"thread_start_us":12539,"threads_started":1}
I20260929 11:37:47.419255 23802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.395258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:46463 (local address 127.0.0.1:55790)
0929 11:37:47.409937 (+ 14679us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.409963 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.410114 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.411485 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.411494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.411517 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:47.411975 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.411989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.414181 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.414190 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.416410 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.416426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.418183 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.418189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.418356 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.418429 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.418579 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.418781 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14531,"mutex_wait_us":26,"thread_start_us":13972,"threads_started":1}
I20260929 11:37:47.420028 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.394926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:34609 (local address 127.0.0.1:51884)
0929 11:37:47.409645 (+ 14719us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.409714 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.409978 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.410798 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.410805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.410838 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:47.411247 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.411267 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.413481 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.413486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.415887 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.415899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.417642 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.417649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.417974 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.419458 (+ 1484us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.419531 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.419671 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14417,"mutex_wait_us":1516,"thread_start_us":15276,"threads_started":1}
I20260929 11:37:47.420118 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.395152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.6.9.194:34609)
0929 11:37:47.408998 (+ 13846us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.409007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.409851 (+ 844us) server_negotiation.cc:407] Connection header received
0929 11:37:47.410312 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.410319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.410545 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.410762 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:47.412211 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.413335 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.416048 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.417427 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.417562 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.417775 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.419619 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.419800 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13669,"thread_start_us":13514,"threads_started":1}
I20260929 11:37:47.422465 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.411084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:34143 (local address 127.0.0.1:34524)
0929 11:37:47.411743 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.411771 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.411917 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.414932 (+ 3015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.414937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.414965 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:47.415393 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.415406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.417198 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.417203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.419272 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.419287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.421599 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.421605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.421778 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.421813 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.421969 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.422113 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":28,"thread_start_us":334,"threads_started":1}
I20260929 11:37:47.422510 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.411139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.6.9.193:34143)
0929 11:37:47.413983 (+ 2844us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.413992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.414025 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:37:47.414129 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.414138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.414424 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.414638 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:47.415590 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.417063 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.419700 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.421359 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.421528 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.421781 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.422013 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.422192 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2631,"thread_start_us":2464,"threads_started":1}
I20260929 11:37:47.423828 23591 tablet_service.cc:1467] Processing CreateTablet for tablet 113842aa7495456496b39b8eaf1d713f (DEFAULT_TABLE table=test-table [id=952352db92f4459fa1579a876c5c992c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:47.423837 23724 tablet_service.cc:1467] Processing CreateTablet for tablet 113842aa7495456496b39b8eaf1d713f (DEFAULT_TABLE table=test-table [id=952352db92f4459fa1579a876c5c992c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:47.425189 23591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113842aa7495456496b39b8eaf1d713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:47.425218 23724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113842aa7495456496b39b8eaf1d713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:47.432140 23457 tablet_service.cc:1467] Processing CreateTablet for tablet 113842aa7495456496b39b8eaf1d713f (DEFAULT_TABLE table=test-table [id=952352db92f4459fa1579a876c5c992c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:47.433816 23457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113842aa7495456496b39b8eaf1d713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:47.441397 23808 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Bootstrap starting.
I20260929 11:37:47.444197 23809 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Bootstrap starting.
I20260929 11:37:47.445309 23808 tablet_bootstrap.cc:654] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:47.446638 23808 log.cc:824] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:47.447752 23809 tablet_bootstrap.cc:654] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:47.448982 23809 log.cc:824] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:47.449369 23808 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: No bootstrap required, opened a new log
I20260929 11:37:47.449652 23808 ts_tablet_manager.cc:1397] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260929 11:37:47.451308 23812 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Bootstrap starting.
I20260929 11:37:47.452087 23809 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: No bootstrap required, opened a new log
I20260929 11:37:47.452359 23809 ts_tablet_manager.cc:1397] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 11:37:47.455096 23812 tablet_bootstrap.cc:654] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:47.456506 23812 log.cc:824] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:47.458290 23808 raft_consensus.cc:348] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.458701 23808 raft_consensus.cc:374] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:47.458815 23808 raft_consensus.cc:729] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0d448258ad40b180bc9f0db0f2b10d, State: Initialized, Role: FOLLOWER
I20260929 11:37:47.459401 23808 consensus_queue.cc:260] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.460108 23812 tablet_bootstrap.cc:492] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: No bootstrap required, opened a new log
I20260929 11:37:47.460410 23812 ts_tablet_manager.cc:1397] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 11:37:47.460624 23656 heartbeater.cc:503] Master 127.6.9.254:44159 was elected leader, sending a full tablet report...
I20260929 11:37:47.461130 23808 ts_tablet_manager.cc:1428] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260929 11:37:47.464257 23809 raft_consensus.cc:348] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.464754 23809 raft_consensus.cc:374] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:47.464854 23809 raft_consensus.cc:729] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67deab5803d24d4cafcf59bca13e7a1c, State: Initialized, Role: FOLLOWER
I20260929 11:37:47.465489 23809 consensus_queue.cc:260] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.469311 23789 heartbeater.cc:503] Master 127.6.9.254:44159 was elected leader, sending a full tablet report...
I20260929 11:37:47.470129 23809 ts_tablet_manager.cc:1428] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260929 11:37:47.470746 23815 raft_consensus.cc:482] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:47.470994 23815 raft_consensus.cc:504] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.471076 23812 raft_consensus.cc:348] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.471544 23812 raft_consensus.cc:374] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:47.471648 23812 raft_consensus.cc:729] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36b6d6d5ce4e9b89673a079492c0a5, State: Initialized, Role: FOLLOWER
I20260929 11:37:47.472295 23812 consensus_queue.cc:260] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.473060 23815 leader_election.cc:288] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb0d448258ad40b180bc9f0db0f2b10d (127.6.9.194:34609), 0a36b6d6d5ce4e9b89673a079492c0a5 (127.6.9.193:34143)
I20260929 11:37:47.474609 23812 ts_tablet_manager.cc:1428] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260929 11:37:47.481003 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.473374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:34609 (local address 127.6.9.195:38343)
0929 11:37:47.474190 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.474209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.474313 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.475372 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.475381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.475407 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:47.475615 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.475626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.476982 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.476988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.478960 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.478976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.480365 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.480370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.480509 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.480535 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.480609 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.480770 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":21}
I20260929 11:37:47.481081 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.474740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:38343 (local address 127.6.9.194:34609)
0929 11:37:47.474925 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.474930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.474948 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:47.475014 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.475019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.475091 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.475176 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:47.475781 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.476813 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.479123 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.480206 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.480345 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.480445 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.480652 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.480811 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:37:47.481997 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.474758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:56233 (local address 127.6.9.193:34143)
0929 11:37:47.474932 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.474938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.475664 (+ 726us) server_negotiation.cc:407] Connection header received
0929 11:37:47.475792 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.475797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.475861 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.475944 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:47.476502 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.477519 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.479828 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.481016 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.481149 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.481245 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.481686 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.481814 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 11:37:47.482156 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.473787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:34143 (local address 127.6.9.195:56233)
0929 11:37:47.475543 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:47.475561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:47.475682 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:47.475970 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:47.475974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:47.475998 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:47.476337 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.476349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.477663 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.477668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.479652 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:47.479666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.481366 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.481371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:47.481528 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.481552 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:37:47.481638 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:37:47.481780 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"mutex_wait_us":3,"thread_start_us":1414,"threads_started":1}
I20260929 11:37:47.482240 23611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113842aa7495456496b39b8eaf1d713f" candidate_uuid: "67deab5803d24d4cafcf59bca13e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" is_pre_election: true
I20260929 11:37:47.482795 23611 raft_consensus.cc:2454] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67deab5803d24d4cafcf59bca13e7a1c in term 0.
I20260929 11:37:47.483089 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113842aa7495456496b39b8eaf1d713f" candidate_uuid: "67deab5803d24d4cafcf59bca13e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" is_pre_election: true
I20260929 11:37:47.483536 23680 leader_election.cc:302] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67deab5803d24d4cafcf59bca13e7a1c, bb0d448258ad40b180bc9f0db0f2b10d; no voters:
I20260929 11:37:47.483644 23477 raft_consensus.cc:2454] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67deab5803d24d4cafcf59bca13e7a1c in term 0.
I20260929 11:37:47.483924 23815 raft_consensus.cc:2781] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:37:47.484078 23815 raft_consensus.cc:482] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:37:47.484169 23815 raft_consensus.cc:3037] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:47.486483 23815 raft_consensus.cc:504] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.487082 23815 leader_election.cc:288] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [CANDIDATE]: Term 1 election: Requested vote from peers bb0d448258ad40b180bc9f0db0f2b10d (127.6.9.194:34609), 0a36b6d6d5ce4e9b89673a079492c0a5 (127.6.9.193:34143)
I20260929 11:37:47.487617 23477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113842aa7495456496b39b8eaf1d713f" candidate_uuid: "67deab5803d24d4cafcf59bca13e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5"
I20260929 11:37:47.487634 23611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113842aa7495456496b39b8eaf1d713f" candidate_uuid: "67deab5803d24d4cafcf59bca13e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0d448258ad40b180bc9f0db0f2b10d"
I20260929 11:37:47.487807 23477 raft_consensus.cc:3037] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:47.487823 23611 raft_consensus.cc:3037] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:47.490144 23611 raft_consensus.cc:2454] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67deab5803d24d4cafcf59bca13e7a1c in term 1.
I20260929 11:37:47.490140 23477 raft_consensus.cc:2454] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67deab5803d24d4cafcf59bca13e7a1c in term 1.
I20260929 11:37:47.490593 23678 leader_election.cc:302] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36b6d6d5ce4e9b89673a079492c0a5, 67deab5803d24d4cafcf59bca13e7a1c; no voters:
I20260929 11:37:47.490950 23815 raft_consensus.cc:2781] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:47.491519 23815 raft_consensus.cc:686] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 1 LEADER]: Becoming Leader. State: Replica: 67deab5803d24d4cafcf59bca13e7a1c, State: Running, Role: LEADER
I20260929 11:37:47.492038 23815 consensus_queue.cc:237] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } }
I20260929 11:37:47.496862 23333 catalog_manager.cc:5495] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 67deab5803d24d4cafcf59bca13e7a1c (127.6.9.195). New cstate: current_term: 1 leader_uuid: "67deab5803d24d4cafcf59bca13e7a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67deab5803d24d4cafcf59bca13e7a1c" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 } health_report { overall_health: UNKNOWN } } }
W20260929 11:37:47.511633 23523 tablet.cc:2369] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:47.528972 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.522897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.6.9.195:46463)
0929 11:37:47.523116 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.523120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.523141 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:37:47.523396 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.523400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.523503 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.523645 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:47.524096 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.524976 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.526680 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.527114 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.527227 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.527772 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:47.528325 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:37:47.528436 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.528576 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.528695 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 11:37:47.529692 23657 tablet.cc:2369] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:47.537352 23477 raft_consensus.cc:1261] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 1 FOLLOWER]: Refusing update from remote peer 67deab5803d24d4cafcf59bca13e7a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:47.537354 23611 raft_consensus.cc:1261] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 1 FOLLOWER]: Refusing update from remote peer 67deab5803d24d4cafcf59bca13e7a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:47.538322 23818 consensus_queue.cc:1035] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0d448258ad40b180bc9f0db0f2b10d" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:47.538664 23815 consensus_queue.cc:1035] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36b6d6d5ce4e9b89673a079492c0a5" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:47.550662 23821 mvcc.cc:205] Tried to move back new op lower bound from 7334632929422315520 to 7334632929251864576. Current Snapshot: MvccSnapshot[applied={T|T < 7334632929422315520}]
I20260929 11:37:47.563423 23332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:47.563709 23332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:47.567783 23332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1af8fccdd2e74b03ac7fa7e690d2032f: Sending DeleteTablet for 3 replicas of tablet 113842aa7495456496b39b8eaf1d713f
I20260929 11:37:47.569067 23724 tablet_service.cc:1514] Processing DeleteTablet for tablet 113842aa7495456496b39b8eaf1d713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:47 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:55790
I20260929 11:37:47.569087 23591 tablet_service.cc:1514] Processing DeleteTablet for tablet 113842aa7495456496b39b8eaf1d713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:47 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:51884
I20260929 11:37:47.569185 23457 tablet_service.cc:1514] Processing DeleteTablet for tablet 113842aa7495456496b39b8eaf1d713f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:47 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:34524
I20260929 11:37:47.569973 23831 tablet_replica.cc:330] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: stopping tablet replica
I20260929 11:37:47.570466 23831 raft_consensus.cc:2229] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:37:47.571223 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23394
I20260929 11:37:47.571272 23831 raft_consensus.cc:2258] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:47.574265 23831 ts_tablet_manager.cc:1905] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:47.581846 23833 tablet_replica.cc:330] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: stopping tablet replica
I20260929 11:37:47.582398 23833 raft_consensus.cc:2229] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:47.582990 23833 raft_consensus.cc:2258] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:47.583253 23831 ts_tablet_manager.cc:1918] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:47.583515 23831 log.cc:1196] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/113842aa7495456496b39b8eaf1d713f
I20260929 11:37:47.584164 23831 ts_tablet_manager.cc:1939] T 113842aa7495456496b39b8eaf1d713f P 67deab5803d24d4cafcf59bca13e7a1c: Deleting consensus metadata
I20260929 11:37:47.586892 23832 tablet_replica.cc:330] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: stopping tablet replica
I20260929 11:37:47.587275 23318 catalog_manager.cc:4841] TS 67deab5803d24d4cafcf59bca13e7a1c (127.6.9.195:46463): tablet 113842aa7495456496b39b8eaf1d713f (table test-table [id=952352db92f4459fa1579a876c5c992c]) successfully deleted
I20260929 11:37:47.587365 23832 raft_consensus.cc:2229] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:47.587908 23832 raft_consensus.cc:2258] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:47.588414 23833 ts_tablet_manager.cc:1905] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:47.589916 23832 ts_tablet_manager.cc:1905] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:37:47 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:37:47.595652 23833 ts_tablet_manager.cc:1918] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:47.595855 23833 log.cc:1196] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/113842aa7495456496b39b8eaf1d713f
I20260929 11:37:47.596405 23833 ts_tablet_manager.cc:1939] T 113842aa7495456496b39b8eaf1d713f P 0a36b6d6d5ce4e9b89673a079492c0a5: Deleting consensus metadata
I20260929 11:37:47.597536 23832 ts_tablet_manager.cc:1918] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:47.597774 23832 log.cc:1196] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/113842aa7495456496b39b8eaf1d713f
I20260929 11:37:47.598402 23832 ts_tablet_manager.cc:1939] T 113842aa7495456496b39b8eaf1d713f P bb0d448258ad40b180bc9f0db0f2b10d: Deleting consensus metadata
I20260929 11:37:47.598937 23318 catalog_manager.cc:4841] TS 0a36b6d6d5ce4e9b89673a079492c0a5 (127.6.9.193:34143): tablet 113842aa7495456496b39b8eaf1d713f (table test-table [id=952352db92f4459fa1579a876c5c992c]) successfully deleted
I20260929 11:37:47.600407 23320 catalog_manager.cc:4841] TS bb0d448258ad40b180bc9f0db0f2b10d (127.6.9.194:34609): tablet 113842aa7495456496b39b8eaf1d713f (table test-table [id=952352db92f4459fa1579a876c5c992c]) successfully deleted
I20260929 11:37:47.603333 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.572223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.6.9.193:34143)
0929 11:37:47.572405 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.572411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.582469 (+ 10058us) server_negotiation.cc:407] Connection header received
0929 11:37:47.583002 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.583009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.583122 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.583204 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:47.585404 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.586361 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.589728 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.590405 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.590510 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.596476 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:47.596493 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:47.596513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:47.596561 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:47.600326 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.601300 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.601316 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.601319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.601475 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.601904 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.601907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.601909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.602259 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:47.602509 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.602861 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.603032 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:37:47.604091 23517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:47.709412 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23394
I20260929 11:37:47.723901 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23528
Sep 29 11:37:47 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:37:47.741070 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.724855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.6.9.194:34609)
0929 11:37:47.725067 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.725074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.725115 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:37:47.725186 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.725192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.725281 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.725389 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:47.726754 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.727886 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.729418 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.730042 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.730174 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.734601 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:47.734620 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:47.734643 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:47.734695 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:47.738313 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.738975 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.738985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.738989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.739142 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.739498 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.739502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.739506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.739988 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:47.740283 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.740447 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.740697 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:37:47.741786 23651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:47.830717 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23528
I20260929 11:37:47.851255 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23661
Sep 29 11:37:47 dist-test-slave-c0l9 krb5kdc[23288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:37:47.868309 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:47.852136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.6.9.195:46463)
0929 11:37:47.852306 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:47.852312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:47.852404 (+ 92us) server_negotiation.cc:407] Connection header received
0929 11:37:47.852594 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:47.852598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:47.852688 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:47.852799 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:47.854076 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.854880 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.856175 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:47.856763 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:47.856909 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:47.861550 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:47.861566 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:47.861582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:47.861625 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:47.864549 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.865805 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.865816 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.865819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.866055 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:47.866461 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:47.866466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:47.866469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:47.866980 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:47.867336 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:47.867655 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 11:37:47.867922 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:37:47.869038 23784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:47.991601 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23661
I20260929 11:37:48.007473 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23301
I20260929 11:37:48.008631 23363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:48.119143 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23301
2026-09-29T11:37:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4234 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T11:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:37:48Z Disabled control of system clock
I20260929 11:37:48.189371 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:38909
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:38909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:37:48.517733 23842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:48.518061 23842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:48.518146 23842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:48.525684 23842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:48.525816 23842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:48.525854 23842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:48.525911 23842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:48.525954 23842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:48.535526 23842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:38909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:38909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:48.537140 23842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:48.539163 23842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:48.551738 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:48.551867 23847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:48.551908 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:48.552882 23842 server_base.cc:1034] running on GCE node
I20260929 11:37:48.553714 23842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:48.555207 23842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:48.556440 23842 hybrid_clock.cc:648] HybridClock initialized: now 1790681868556381 us; error 51 us; skew 500 ppm
I20260929 11:37:48.556878 23842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:48.559599 23842 webserver.cc:460] Webserver started at http://127.6.9.254:42305/ using document root <none> and password file <none>
I20260929 11:37:48.560387 23842 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:48.560489 23842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:48.560889 23842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:48.564034 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9"
format_stamp: "Formatted at 2026-09-29 11:37:48 on dist-test-slave-c0l9"
server_key: "dc043875efc595eadd05c7a784cbbf03"
server_key_iv: "095f4ea62bae64c451f3ba99b937763c"
server_key_version: "encryptionkey@0"
I20260929 11:37:48.564996 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9"
format_stamp: "Formatted at 2026-09-29 11:37:48 on dist-test-slave-c0l9"
server_key: "dc043875efc595eadd05c7a784cbbf03"
server_key_iv: "095f4ea62bae64c451f3ba99b937763c"
server_key_version: "encryptionkey@0"
I20260929 11:37:48.570804 23842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:37:48.574514 23856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:48.575976 23842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:37:48.576169 23842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9"
format_stamp: "Formatted at 2026-09-29 11:37:48 on dist-test-slave-c0l9"
server_key: "dc043875efc595eadd05c7a784cbbf03"
server_key_iv: "095f4ea62bae64c451f3ba99b937763c"
server_key_version: "encryptionkey@0"
I20260929 11:37:48.576413 23842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:48.601612 23842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:48.602546 23842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:48.602818 23842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:48.619577 23842 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:38909
I20260929 11:37:48.619598 23907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:38909 every 8 connection(s)
I20260929 11:37:48.621062 23842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:48.622798 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23842
I20260929 11:37:48.622963 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:37:48.623368 6183 external_mini_cluster.cc:1428] Setting key f62e125fc5efbfc0f72fed8daee19529
I20260929 11:37:48.625324 23908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:48.639364 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:48.625275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.6.9.254:38909)
0929 11:37:48.626322 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:48.626336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:48.626387 (+ 51us) server_negotiation.cc:407] Connection header received
0929 11:37:48.628465 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:48.628496 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:48.629188 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:48.629700 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:48.630250 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:48.632290 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:48.633834 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:48.634845 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:48.635027 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:48.636272 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:48.636284 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:48.636305 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:48.637153 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:48.637261 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:48.637292 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:48.637419 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:48.637673 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 11:37:48.638053 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":331,"threads_started":1}
I20260929 11:37:48.640085 23908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: Bootstrap starting.
I20260929 11:37:48.643407 23908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:48.644452 23908 log.cc:824] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:48.647516 23908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: No bootstrap required, opened a new log
I20260929 11:37:48.655297 23908 raft_consensus.cc:348] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } }
I20260929 11:37:48.655705 23908 raft_consensus.cc:374] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:48.655809 23908 raft_consensus.cc:729] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b944ac7a64cb48c5a61bc406cc8ba9a9, State: Initialized, Role: FOLLOWER
I20260929 11:37:48.656411 23908 consensus_queue.cc:260] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } }
I20260929 11:37:48.656656 23908 raft_consensus.cc:388] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:48.656844 23908 raft_consensus.cc:482] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:48.657008 23908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:48.659375 23908 raft_consensus.cc:504] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } }
I20260929 11:37:48.659940 23908 leader_election.cc:302] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b944ac7a64cb48c5a61bc406cc8ba9a9; no voters:
I20260929 11:37:48.660564 23908 leader_election.cc:288] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:37:48.660761 23913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:48.661542 23913 raft_consensus.cc:686] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [term 1 LEADER]: Becoming Leader. State: Replica: b944ac7a64cb48c5a61bc406cc8ba9a9, State: Running, Role: LEADER
I20260929 11:37:48.662160 23913 consensus_queue.cc:237] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } }
I20260929 11:37:48.662942 23908 sys_catalog.cc:564] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:48.665289 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } } }
I20260929 11:37:48.665701 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:48.665374 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b944ac7a64cb48c5a61bc406cc8ba9a9. Latest consensus state: current_term: 1 leader_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b944ac7a64cb48c5a61bc406cc8ba9a9" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 38909 } } }
I20260929 11:37:48.668737 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:48.674067 23921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:48.679880 23921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:48.689452 23921 catalog_manager.cc:1347] Generated new cluster ID: ace3e44fc93745b8992825d805927e1d
I20260929 11:37:48.689580 23921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:48.703666 23921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:37:48.705034 23921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:48.715271 23921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: Generated new TSK 0
I20260929 11:37:48.716180 23921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:37:48.728237 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--builtin_ntp_servers=127.6.9.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:37:49.080279 23932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:49.080583 23932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:49.080688 23932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:49.088183 23932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:49.088320 23932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:49.088455 23932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:49.097909 23932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:49.099346 23932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:49.101751 23932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:49.115074 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:49.115103 23940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:49.115034 23938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:49.116400 23932 server_base.cc:1034] running on GCE node
I20260929 11:37:49.117116 23932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:49.118139 23932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:49.119386 23932 hybrid_clock.cc:648] HybridClock initialized: now 1790681869119331 us; error 47 us; skew 500 ppm
I20260929 11:37:49.119728 23932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:49.122257 23932 webserver.cc:460] Webserver started at http://127.6.9.193:40703/ using document root <none> and password file <none>
I20260929 11:37:49.122978 23932 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:49.123077 23932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:49.123414 23932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:49.126461 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "422f5e9ff3574255adedfd7cc8610587"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "2da45d8017c6755e312cecd78d42eef4"
server_key_iv: "68cfcfc913d51ed71762b4cc265958c7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.127272 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "422f5e9ff3574255adedfd7cc8610587"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "2da45d8017c6755e312cecd78d42eef4"
server_key_iv: "68cfcfc913d51ed71762b4cc265958c7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.132529 23932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:37:49.136148 23946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.137686 23932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:37:49.137874 23932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "422f5e9ff3574255adedfd7cc8610587"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "2da45d8017c6755e312cecd78d42eef4"
server_key_iv: "68cfcfc913d51ed71762b4cc265958c7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.138092 23932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:49.156828 23932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:49.157699 23932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:49.157968 23932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:49.158941 23932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:49.160398 23932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:49.160485 23932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.160593 23932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:49.160691 23932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.191816 23932 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:33733
I20260929 11:37:49.191862 24058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:33733 every 8 connection(s)
I20260929 11:37:49.193157 23932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:49.196038 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 23932
I20260929 11:37:49.196218 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:37:49.196789 6183 external_mini_cluster.cc:1428] Setting key 078e77aa3dec5f741b06c6fda768c4de
I20260929 11:37:49.199474 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--builtin_ntp_servers=127.6.9.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:37:49.206418 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:49.195553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:54389 (local address 127.6.9.254:38909)
0929 11:37:49.195992 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:49.195997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:49.197299 (+ 1302us) server_negotiation.cc:407] Connection header received
0929 11:37:49.199252 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:49.199257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:49.199337 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:49.199427 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:49.200481 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.201239 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.203319 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.203917 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.204048 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:49.205562 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:49.205568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:49.205572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:49.205596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:49.205646 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:49.205661 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:49.205743 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:49.205945 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 11:37:49.206059 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260929 11:37:49.207513 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:49.196125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38909 (local address 127.6.9.193:54389)
0929 11:37:49.197142 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:49.197178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:49.199013 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:49.199672 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:49.199683 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:49.199770 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:49.200321 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:49.200338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.201416 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.201421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:49.203144 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:49.203160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.204136 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.204141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:49.204418 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:49.205185 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:49.205188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:49.205240 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:49.205352 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:49.205751 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:49.205759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:49.205766 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:37:49.205928 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 11:37:49.206391 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":317,"threads_started":1}
I20260929 11:37:49.209061 24059 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38909
I20260929 11:37:49.209342 24059 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:49.210091 24059 heartbeater.cc:511] Master 127.6.9.254:38909 requested a full tablet report, sending...
I20260929 11:37:49.212127 23873 ts_manager.cc:194] Registered new tserver with Master: 422f5e9ff3574255adedfd7cc8610587 (127.6.9.193:33733)
I20260929 11:37:49.213826 23873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.193:54389
W20260929 11:37:49.532147 24063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:49.532537 24063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:49.532851 24063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:49.540455 24063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:49.540597 24063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:49.540788 24063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:49.550535 24063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:49.551855 24063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:49.553984 24063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:49.566496 24068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:49.566466 24071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:49.566529 24069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:49.567256 24063 server_base.cc:1034] running on GCE node
I20260929 11:37:49.567865 24063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:49.568838 24063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:49.570052 24063 hybrid_clock.cc:648] HybridClock initialized: now 1790681869570000 us; error 47 us; skew 500 ppm
I20260929 11:37:49.570354 24063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:49.572515 24063 webserver.cc:460] Webserver started at http://127.6.9.194:36435/ using document root <none> and password file <none>
I20260929 11:37:49.573251 24063 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:49.573349 24063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:49.573652 24063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:49.576493 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "bac920f09c0949658ac80d160a80f55c"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "f2df03a01f70cffbba4ac30c123ab40e"
server_key_iv: "5ae81f68b3139951aa5e8f404462b8f7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.577306 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "bac920f09c0949658ac80d160a80f55c"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "f2df03a01f70cffbba4ac30c123ab40e"
server_key_iv: "5ae81f68b3139951aa5e8f404462b8f7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.582408 24063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:37:49.585964 24077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.587296 24063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:49.587456 24063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "bac920f09c0949658ac80d160a80f55c"
format_stamp: "Formatted at 2026-09-29 11:37:49 on dist-test-slave-c0l9"
server_key: "f2df03a01f70cffbba4ac30c123ab40e"
server_key_iv: "5ae81f68b3139951aa5e8f404462b8f7"
server_key_version: "encryptionkey@0"
I20260929 11:37:49.587661 24063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:49.604548 24063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:49.605422 24063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:49.605679 24063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:49.606635 24063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:49.608268 24063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:49.608366 24063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.608477 24063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:49.608546 24063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:49.642014 24063 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:37519
I20260929 11:37:49.642073 24189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:37519 every 8 connection(s)
I20260929 11:37:49.643368 24063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:49.645089 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24063
I20260929 11:37:49.645272 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:37:49.645742 6183 external_mini_cluster.cc:1428] Setting key d8f5298a355ae5d19060e92638109e24
I20260929 11:37:49.652995 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--builtin_ntp_servers=127.6.9.212:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:37:49.657205 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:49.646119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:41731 (local address 127.6.9.254:38909)
0929 11:37:49.646330 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:49.646337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:49.647927 (+ 1590us) server_negotiation.cc:407] Connection header received
0929 11:37:49.650457 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:49.650463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:49.650562 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:49.650682 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:49.652001 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.652845 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.654509 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.654909 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.655020 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:49.656369 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:49.656374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:49.656377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:49.656396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:49.656428 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:49.656440 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:49.656515 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:49.656780 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 11:37:49.656886 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:37:49.658352 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:49.646637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38909 (local address 127.6.9.194:41731)
0929 11:37:49.647745 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:49.647795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:49.650118 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:49.651000 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:49.651039 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:49.651153 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:49.651832 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:49.651858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.652988 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.652994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:49.654355 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:49.654368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:49.655134 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:49.655142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:49.655372 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:49.656045 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:49.656047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:49.656089 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:49.656171 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:49.656528 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:49.656539 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:49.656549 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:37:49.656749 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 11:37:49.657204 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":314,"threads_started":1}
I20260929 11:37:49.659698 24190 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38909
I20260929 11:37:49.660017 24190 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:49.660981 24190 heartbeater.cc:511] Master 127.6.9.254:38909 requested a full tablet report, sending...
I20260929 11:37:49.662387 23873 ts_manager.cc:194] Registered new tserver with Master: bac920f09c0949658ac80d160a80f55c (127.6.9.194:37519)
I20260929 11:37:49.663228 23873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.194:41731
W20260929 11:37:49.982446 24193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:49.982758 24193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:49.982882 24193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:49.990307 24193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:49.990451 24193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:49.990626 24193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:50.000244 24193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:38909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:50.001662 24193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:50.003489 24193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:50.016932 24198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:50.017011 24199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:50.016965 24201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:50.017956 24193 server_base.cc:1034] running on GCE node
I20260929 11:37:50.018579 24193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:50.019591 24193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:50.020757 24193 hybrid_clock.cc:648] HybridClock initialized: now 1790681870020690 us; error 66 us; skew 500 ppm
I20260929 11:37:50.021098 24193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:50.023516 24193 webserver.cc:460] Webserver started at http://127.6.9.195:39157/ using document root <none> and password file <none>
I20260929 11:37:50.024233 24193 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:50.024343 24193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:50.024689 24193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:50.027670 24193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "0dea118926c84a08923fc7f20ac3afed"
format_stamp: "Formatted at 2026-09-29 11:37:50 on dist-test-slave-c0l9"
server_key: "d00a041fcb3c9fdfb5368f00f7e2c373"
server_key_iv: "6e66215b376ffc6eab7cd7cd7a7f3bcd"
server_key_version: "encryptionkey@0"
I20260929 11:37:50.028462 24193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "0dea118926c84a08923fc7f20ac3afed"
format_stamp: "Formatted at 2026-09-29 11:37:50 on dist-test-slave-c0l9"
server_key: "d00a041fcb3c9fdfb5368f00f7e2c373"
server_key_iv: "6e66215b376ffc6eab7cd7cd7a7f3bcd"
server_key_version: "encryptionkey@0"
I20260929 11:37:50.034247 24193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:37:50.038095 24207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:50.039515 24193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:50.039674 24193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "0dea118926c84a08923fc7f20ac3afed"
format_stamp: "Formatted at 2026-09-29 11:37:50 on dist-test-slave-c0l9"
server_key: "d00a041fcb3c9fdfb5368f00f7e2c373"
server_key_iv: "6e66215b376ffc6eab7cd7cd7a7f3bcd"
server_key_version: "encryptionkey@0"
I20260929 11:37:50.039886 24193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:50.082978 24193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:50.083887 24193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:50.084178 24193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:50.085302 24193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:50.086998 24193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:50.087095 24193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:50.087199 24193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:50.087266 24193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:50.117477 24193 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:38009
I20260929 11:37:50.117519 24319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:38009 every 8 connection(s)
I20260929 11:37:50.118738 24193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:50.128357 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24193
I20260929 11:37:50.128552 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:37:50.129220 6183 external_mini_cluster.cc:1428] Setting key fa202e35e116b5f59f1ca52addc8e959
I20260929 11:37:50.131699 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.121227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:34063 (local address 127.6.9.254:38909)
0929 11:37:50.121432 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.121438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.122842 (+ 1404us) server_negotiation.cc:407] Connection header received
0929 11:37:50.124625 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.124630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.124735 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.124823 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:50.125926 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.126664 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.128264 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.128965 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.129117 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.130744 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:50.130751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:50.130755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:50.130802 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:50.130846 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:50.130864 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:50.130958 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.131247 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.131371 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:37:50.132951 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.121797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:38909 (local address 127.6.9.195:34063)
0929 11:37:50.122685 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.122721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.124359 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.125058 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.125067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.125159 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:50.125733 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.125750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.126816 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.126819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.128079 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.128091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.129223 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.129229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.129519 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.130359 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:50.130363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:50.130431 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:50.130544 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:50.130969 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:50.130982 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:50.130990 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.131235 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.131676 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":274,"threads_started":1}
I20260929 11:37:50.134388 24320 heartbeater.cc:347] Connected to a master server at 127.6.9.254:38909
I20260929 11:37:50.134722 24320 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:50.135550 24320 heartbeater.cc:511] Master 127.6.9.254:38909 requested a full tablet report, sending...
I20260929 11:37:50.137190 23873 ts_manager.cc:194] Registered new tserver with Master: 0dea118926c84a08923fc7f20ac3afed (127.6.9.195:38009)
I20260929 11:37:50.138074 23873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.195:34063
I20260929 11:37:50.145453 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:37:50.156170 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.149828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.6.9.254:38909)
0929 11:37:50.150084 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.150090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.150461 (+ 371us) server_negotiation.cc:407] Connection header received
0929 11:37:50.150604 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.150608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.150686 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.150797 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:50.151356 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.152060 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.153588 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.154034 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.154149 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.155358 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:50.155363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:50.155366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:50.155388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:50.155423 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:50.155444 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:50.155515 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.155739 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.155856 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:37:50.161101 23873 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:37:50.168498 23873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:37:50.195024 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.181894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:38009 (local address 127.0.0.1:32904)
0929 11:37:50.183969 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.183987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.184119 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.185031 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.185041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.185085 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:50.185488 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.185501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.187576 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.187582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.190187 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.190201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.192379 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.194026 (+ 1647us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.194286 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.194361 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.194494 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.194690 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1925,"thread_start_us":722,"threads_started":1}
I20260929 11:37:50.196135 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.182113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.6.9.195:38009)
0929 11:37:50.182859 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.182876 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.184079 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 11:37:50.184262 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.184271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.184492 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.184711 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:50.185707 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.187426 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.190425 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.192047 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.192185 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.193987 (+ 1802us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.195602 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.195789 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":436,"threads_started":1}
I20260929 11:37:50.200402 24255 tablet_service.cc:1467] Processing CreateTablet for tablet 1800366183c649689d0be60c52d6cd4a (DEFAULT_TABLE table=test-table [id=b6ef94ea401648698ed80a82c4626c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:50.202075 24255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1800366183c649689d0be60c52d6cd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:50.203146 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.181402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48546)
0929 11:37:50.182945 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.182976 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.183134 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.189874 (+ 6740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.189880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.189910 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:50.190379 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.190391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.192590 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.192598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.194883 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.194900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.202425 (+ 7525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.202430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.202568 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.202593 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.202686 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.202840 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"thread_start_us":19451,"threads_started":1}
I20260929 11:37:50.203328 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.181535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.6.9.193:33733)
0929 11:37:50.182324 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.182333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.183082 (+ 749us) server_negotiation.cc:407] Connection header received
0929 11:37:50.183323 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.183332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.183588 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.183790 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:50.190562 (+ 6772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.192331 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.195238 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.196789 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.196928 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.197221 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.202780 (+ 5559us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.202986 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":746,"threads_started":1}
I20260929 11:37:50.205303 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.183402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:37519 (local address 127.0.0.1:34736)
0929 11:37:50.195310 (+ 11908us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.195331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.195431 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.198625 (+ 3194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.198630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.198658 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:50.198851 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.198861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.200951 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.200957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.202784 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.202800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.204561 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.204568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.204787 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.204824 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.204888 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.205020 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11785,"thread_start_us":258,"threads_started":1}
I20260929 11:37:50.206509 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.184032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.6.9.194:37519)
0929 11:37:50.197772 (+ 13740us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.197780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.197811 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:37:50.197921 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.197928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.198188 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.199039 (+ 851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:50.199251 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.200764 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.203004 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.204366 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.205662 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.205917 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.205997 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.206167 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13565,"thread_start_us":14308,"threads_started":1}
I20260929 11:37:50.207399 23994 tablet_service.cc:1467] Processing CreateTablet for tablet 1800366183c649689d0be60c52d6cd4a (DEFAULT_TABLE table=test-table [id=b6ef94ea401648698ed80a82c4626c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:50.208619 23994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1800366183c649689d0be60c52d6cd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:50.210121 24125 tablet_service.cc:1467] Processing CreateTablet for tablet 1800366183c649689d0be60c52d6cd4a (DEFAULT_TABLE table=test-table [id=b6ef94ea401648698ed80a82c4626c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:50.211608 24125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1800366183c649689d0be60c52d6cd4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:50.216467 24338 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Bootstrap starting.
I20260929 11:37:50.217129 24059 heartbeater.cc:503] Master 127.6.9.254:38909 was elected leader, sending a full tablet report...
I20260929 11:37:50.220117 24338 tablet_bootstrap.cc:654] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:50.222007 24339 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Bootstrap starting.
I20260929 11:37:50.222100 24338 log.cc:824] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:50.225493 24339 tablet_bootstrap.cc:654] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:50.225878 24338 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: No bootstrap required, opened a new log
I20260929 11:37:50.226246 24338 ts_tablet_manager.cc:1397] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 11:37:50.226490 24341 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Bootstrap starting.
I20260929 11:37:50.226706 24339 log.cc:824] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:50.229326 24339 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: No bootstrap required, opened a new log
I20260929 11:37:50.229562 24339 ts_tablet_manager.cc:1397] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 11:37:50.230084 24341 tablet_bootstrap.cc:654] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:50.231280 24341 log.cc:824] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:50.234982 24341 tablet_bootstrap.cc:492] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: No bootstrap required, opened a new log
I20260929 11:37:50.235347 24341 ts_tablet_manager.cc:1397] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:37:50.236318 24338 raft_consensus.cc:348] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.236953 24338 raft_consensus.cc:374] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:50.237095 24338 raft_consensus.cc:729] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dea118926c84a08923fc7f20ac3afed, State: Initialized, Role: FOLLOWER
I20260929 11:37:50.237905 24338 consensus_queue.cc:260] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.238260 24339 raft_consensus.cc:348] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.238766 24339 raft_consensus.cc:374] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:50.238916 24339 raft_consensus.cc:729] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422f5e9ff3574255adedfd7cc8610587, State: Initialized, Role: FOLLOWER
I20260929 11:37:50.239320 24320 heartbeater.cc:503] Master 127.6.9.254:38909 was elected leader, sending a full tablet report...
I20260929 11:37:50.239531 24339 consensus_queue.cc:260] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.239971 24338 ts_tablet_manager.cc:1428] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 11:37:50.241099 24339 ts_tablet_manager.cc:1428] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260929 11:37:50.246797 24341 raft_consensus.cc:348] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.247192 24341 raft_consensus.cc:374] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:50.247328 24341 raft_consensus.cc:729] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac920f09c0949658ac80d160a80f55c, State: Initialized, Role: FOLLOWER
I20260929 11:37:50.247861 24341 consensus_queue.cc:260] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.248857 24190 heartbeater.cc:503] Master 127.6.9.254:38909 was elected leader, sending a full tablet report...
I20260929 11:37:50.249325 24341 ts_tablet_manager.cc:1428] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260929 11:37:50.262251 24346 raft_consensus.cc:482] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:50.262555 24346 raft_consensus.cc:504] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.264201 24346 leader_election.cc:288] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422f5e9ff3574255adedfd7cc8610587 (127.6.9.193:33733), 0dea118926c84a08923fc7f20ac3afed (127.6.9.195:38009)
I20260929 11:37:50.272943 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.264800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:44825 (local address 127.6.9.195:38009)
0929 11:37:50.265013 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.265020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.265640 (+ 620us) server_negotiation.cc:407] Connection header received
0929 11:37:50.265823 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.265829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.265919 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.266008 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:50.266685 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.267780 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.270662 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.271854 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.271985 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.272088 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.272507 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.272641 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:37:50.272918 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.265006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:38187 (local address 127.6.9.193:33733)
0929 11:37:50.265232 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.265239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.265872 (+ 633us) server_negotiation.cc:407] Connection header received
0929 11:37:50.266020 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.266026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.266111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.266185 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:50.266876 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.267875 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.270638 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.271852 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.271985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.272088 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.272507 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.272641 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:37:50.273048 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.264480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:38009 (local address 127.6.9.194:44825)
0929 11:37:50.265527 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.265543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.265691 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.266049 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.266053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.266076 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:50.266531 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.266550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.267922 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.267927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.270412 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.270421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.272095 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.272101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.272305 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.272370 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.272465 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.272640 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":137,"thread_start_us":712,"threads_started":1}
I20260929 11:37:50.273425 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.264535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.6.9.194:38187)
0929 11:37:50.265759 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.265775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:50.265897 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:50.266233 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:50.266238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:50.266263 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:50.266734 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.266744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.268008 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.268012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.270400 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:50.270417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.272072 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.272083 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:50.272305 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.272366 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 11:37:50.272465 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:37:50.272640 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":226,"thread_start_us":892,"threads_started":1}
I20260929 11:37:50.274111 24275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800366183c649689d0be60c52d6cd4a" candidate_uuid: "bac920f09c0949658ac80d160a80f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dea118926c84a08923fc7f20ac3afed" is_pre_election: true
I20260929 11:37:50.274421 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800366183c649689d0be60c52d6cd4a" candidate_uuid: "bac920f09c0949658ac80d160a80f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422f5e9ff3574255adedfd7cc8610587" is_pre_election: true
I20260929 11:37:50.274783 24275 raft_consensus.cc:2454] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac920f09c0949658ac80d160a80f55c in term 0.
I20260929 11:37:50.275023 24014 raft_consensus.cc:2454] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bac920f09c0949658ac80d160a80f55c in term 0.
I20260929 11:37:50.275610 24080 leader_election.cc:302] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dea118926c84a08923fc7f20ac3afed, bac920f09c0949658ac80d160a80f55c; no voters:
I20260929 11:37:50.276093 24346 raft_consensus.cc:2781] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:37:50.276243 24346 raft_consensus.cc:482] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:37:50.276343 24346 raft_consensus.cc:3037] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:50.278573 24346 raft_consensus.cc:504] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.279156 24346 leader_election.cc:288] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [CANDIDATE]: Term 1 election: Requested vote from peers 422f5e9ff3574255adedfd7cc8610587 (127.6.9.193:33733), 0dea118926c84a08923fc7f20ac3afed (127.6.9.195:38009)
I20260929 11:37:50.279763 24275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800366183c649689d0be60c52d6cd4a" candidate_uuid: "bac920f09c0949658ac80d160a80f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dea118926c84a08923fc7f20ac3afed"
I20260929 11:37:50.279763 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800366183c649689d0be60c52d6cd4a" candidate_uuid: "bac920f09c0949658ac80d160a80f55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422f5e9ff3574255adedfd7cc8610587"
I20260929 11:37:50.280019 24014 raft_consensus.cc:3037] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:50.280010 24275 raft_consensus.cc:3037] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:50.283242 24014 raft_consensus.cc:2454] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac920f09c0949658ac80d160a80f55c in term 1.
I20260929 11:37:50.283242 24275 raft_consensus.cc:2454] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bac920f09c0949658ac80d160a80f55c in term 1.
I20260929 11:37:50.283730 24080 leader_election.cc:302] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dea118926c84a08923fc7f20ac3afed, bac920f09c0949658ac80d160a80f55c; no voters:
I20260929 11:37:50.284052 24346 raft_consensus.cc:2781] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:50.284469 24346 raft_consensus.cc:686] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 1 LEADER]: Becoming Leader. State: Replica: bac920f09c0949658ac80d160a80f55c, State: Running, Role: LEADER
I20260929 11:37:50.285130 24346 consensus_queue.cc:237] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } }
I20260929 11:37:50.289892 23871 catalog_manager.cc:5495] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c reported cstate change: term changed from 0 to 1, leader changed from <none> to bac920f09c0949658ac80d160a80f55c (127.6.9.194). New cstate: current_term: 1 leader_uuid: "bac920f09c0949658ac80d160a80f55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bac920f09c0949658ac80d160a80f55c" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 37519 } health_report { overall_health: HEALTHY } } }
I20260929 11:37:50.317062 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.312056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.6.9.194:37519)
0929 11:37:50.312254 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.312259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.312278 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:37:50.312366 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.312370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.312456 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.312579 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:50.312954 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.313665 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.315007 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.315411 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.315512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.315989 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:50.316497 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:37:50.316611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.316765 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.316848 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:37:50.324983 24275 raft_consensus.cc:1261] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 1 FOLLOWER]: Refusing update from remote peer bac920f09c0949658ac80d160a80f55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:50.324988 24014 raft_consensus.cc:1261] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 1 FOLLOWER]: Refusing update from remote peer bac920f09c0949658ac80d160a80f55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:50.325832 24349 consensus_queue.cc:1035] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422f5e9ff3574255adedfd7cc8610587" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:50.326151 24346 consensus_queue.cc:1035] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dea118926c84a08923fc7f20ac3afed" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 38009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:50.354789 23871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:50.355114 23871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:50.358807 23871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b944ac7a64cb48c5a61bc406cc8ba9a9: Sending DeleteTablet for 3 replicas of tablet 1800366183c649689d0be60c52d6cd4a
I20260929 11:37:50.360095 24125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1800366183c649689d0be60c52d6cd4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:50 UTC) from {username='slave'} at 127.0.0.1:34736
I20260929 11:37:50.360062 23994 tablet_service.cc:1514] Processing DeleteTablet for tablet 1800366183c649689d0be60c52d6cd4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:50 UTC) from {username='slave'} at 127.0.0.1:48546
I20260929 11:37:50.360289 24255 tablet_service.cc:1514] Processing DeleteTablet for tablet 1800366183c649689d0be60c52d6cd4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:50 UTC) from {username='slave'} at 127.0.0.1:32904
I20260929 11:37:50.362118 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23932
I20260929 11:37:50.381313 24363 tablet_replica.cc:330] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: stopping tablet replica
I20260929 11:37:50.381541 24364 tablet_replica.cc:330] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: stopping tablet replica
I20260929 11:37:50.381691 24362 tablet_replica.cc:330] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: stopping tablet replica
I20260929 11:37:50.381994 24364 raft_consensus.cc:2229] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:50.381992 24363 raft_consensus.cc:2229] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:50.382134 24362 raft_consensus.cc:2229] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:37:50.382539 24363 raft_consensus.cc:2258] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:50.382539 24364 raft_consensus.cc:2258] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:50.382800 24362 raft_consensus.cc:2258] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:50.384732 24364 ts_tablet_manager.cc:1905] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:50.385120 24362 ts_tablet_manager.cc:1905] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:50.385136 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.364202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.6.9.193:33733)
0929 11:37:50.364410 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.364416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.368623 (+ 4207us) server_negotiation.cc:407] Connection header received
0929 11:37:50.369441 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.369448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.369587 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.369665 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:50.372259 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.373839 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.377850 (+ 4011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.378582 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.378685 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.383313 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:50.383325 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:50.383344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:50.384132 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:50.384197 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:50.384231 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:50.384325 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.384600 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.384768 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:37:50.385136 24363 ts_tablet_manager.cc:1905] T 1800366183c649689d0be60c52d6cd4a P 422f5e9ff3574255adedfd7cc8610587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:50.386210 24054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:50.392830 24364 ts_tablet_manager.cc:1918] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:50.393019 24364 log.cc:1196] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/1800366183c649689d0be60c52d6cd4a
I20260929 11:37:50.393563 24364 ts_tablet_manager.cc:1939] T 1800366183c649689d0be60c52d6cd4a P 0dea118926c84a08923fc7f20ac3afed: Deleting consensus metadata
I20260929 11:37:50.393777 24362 ts_tablet_manager.cc:1918] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:50.394021 24362 log.cc:1196] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/1800366183c649689d0be60c52d6cd4a
I20260929 11:37:50.394614 24362 ts_tablet_manager.cc:1939] T 1800366183c649689d0be60c52d6cd4a P bac920f09c0949658ac80d160a80f55c: Deleting consensus metadata
I20260929 11:37:50.396246 23859 catalog_manager.cc:4841] TS 0dea118926c84a08923fc7f20ac3afed (127.6.9.195:38009): tablet 1800366183c649689d0be60c52d6cd4a (table test-table [id=b6ef94ea401648698ed80a82c4626c7e]) successfully deleted
I20260929 11:37:50.397424 23859 catalog_manager.cc:4841] TS bac920f09c0949658ac80d160a80f55c (127.6.9.194:37519): tablet 1800366183c649689d0be60c52d6cd4a (table test-table [id=b6ef94ea401648698ed80a82c4626c7e]) successfully deleted
I20260929 11:37:50.484468 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23932
I20260929 11:37:50.499282 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24063
W20260929 11:37:50.499475 23857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.9.193:33733 (error 108)
I20260929 11:37:50.502316 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.501738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48562)
0929 11:37:50.501963 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.502094 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 11:37:50.502599 23857 catalog_manager.cc:4584] TS 422f5e9ff3574255adedfd7cc8610587 (127.6.9.193:33733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1800366183c649689d0be60c52d6cd4a: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
I20260929 11:37:50.508137 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.500156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.6.9.194:37519)
0929 11:37:50.500354 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.500361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.500479 (+ 118us) server_negotiation.cc:407] Connection header received
0929 11:37:50.500709 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.500715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.500837 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.500947 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:50.501629 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.502727 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.504818 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.505254 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.505370 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.506695 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:50.506704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:50.506719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:50.507361 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:50.507410 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:50.507434 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:50.507519 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.507699 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.507789 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:37:50.508857 24185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:50.556268 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.555798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48574)
0929 11:37:50.556019 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.556118 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:37:50.557425 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.557025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48582)
0929 11:37:50.557189 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.557266 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:37:50.618237 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.617725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48584)
0929 11:37:50.617980 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.618081 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 11:37:50.619454 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.618947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:48594)
0929 11:37:50.619138 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.619271 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 11:37:50.622020 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24063
I20260929 11:37:50.637744 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24193
I20260929 11:37:50.644874 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.638520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.6.9.195:38009)
0929 11:37:50.638696 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:50.638701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:50.638722 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:37:50.638921 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:50.638925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:50.639008 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:50.639096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:50.639527 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.640244 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.641686 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:50.642071 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:50.642183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:50.643277 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:50.643286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:50.643302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:50.643870 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:50.643920 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:50.643940 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:50.644039 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:50.644190 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 11:37:50.644307 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:37:50.645563 24315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:50.700230 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.699737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:59524)
0929 11:37:50.699973 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.700074 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 11:37:50.701658 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:50.701149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:33733 (local address 127.0.0.1:59538)
0929 11:37:50.701408 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:50.701504 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 11:37:50.734781 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24193
I20260929 11:37:50.748984 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 23842
I20260929 11:37:50.750247 23903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:50.859905 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 23842
2026-09-29T11:37:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2740 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:37:50 dist-test-slave-c0l9 krb5kdc[24373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:37:50 dist-test-slave-c0l9 krb5kdc[24373](info): set up 2 sockets
Sep 29 11:37:50 dist-test-slave-c0l9 krb5kdc[24373](info): commencing operation
krb5kdc: starting...
W20260929 11:37:51.892880 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:37:52 dist-test-slave-c0l9 krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:37:52.137143 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44123
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:52.470232 24386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:52.470580 24386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:52.470683 24386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:52.478139 24386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:52.478276 24386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:52.478345 24386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:52.478430 24386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:52.478487 24386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:52.488830 24386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:52.490144 24386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:52.491387 24386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:52.503021 24392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:52.503247 24394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:52.503026 24391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:52.503803 24386 server_base.cc:1034] running on GCE node
I20260929 11:37:52.504638 24386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:52.506104 24386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:52.507329 24386 hybrid_clock.cc:648] HybridClock initialized: now 1790681872507271 us; error 51 us; skew 500 ppm
I20260929 11:37:52.507678 24386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:52 dist-test-slave-c0l9 krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:52.515826 24386 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:37:52.517522 24386 webserver.cc:460] Webserver started at http://127.6.9.254:43889/ using document root <none> and password file <none>
I20260929 11:37:52.518251 24386 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:52.518376 24386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:52.518677 24386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:52.521817 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "749f1e3fe286491dad52eb6dc9d3a4c5"
format_stamp: "Formatted at 2026-09-29 11:37:52 on dist-test-slave-c0l9"
server_key: "236c6613de0868425ded1057b43dcf2c"
server_key_iv: "5b1980c3759bb538014067f366bfc67d"
server_key_version: "encryptionkey@0"
I20260929 11:37:52.522565 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "749f1e3fe286491dad52eb6dc9d3a4c5"
format_stamp: "Formatted at 2026-09-29 11:37:52 on dist-test-slave-c0l9"
server_key: "236c6613de0868425ded1057b43dcf2c"
server_key_iv: "5b1980c3759bb538014067f366bfc67d"
server_key_version: "encryptionkey@0"
I20260929 11:37:52.527720 24386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:37:52.531275 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:52.532579 24386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:37:52.532783 24386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "749f1e3fe286491dad52eb6dc9d3a4c5"
format_stamp: "Formatted at 2026-09-29 11:37:52 on dist-test-slave-c0l9"
server_key: "236c6613de0868425ded1057b43dcf2c"
server_key_iv: "5b1980c3759bb538014067f366bfc67d"
server_key_version: "encryptionkey@0"
I20260929 11:37:52.532967 24386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:52.549866 24386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:52.553918 24386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:52.554183 24386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:52.571671 24386 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:44123
I20260929 11:37:52.571704 24452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:44123 every 8 connection(s)
I20260929 11:37:52.573063 24386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:52.576874 24453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:52.580432 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24386
I20260929 11:37:52.580641 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:37:52.581239 6183 external_mini_cluster.cc:1428] Setting key 09464c39f422426877c73a7d9e17e506
I20260929 11:37:52.589493 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: Bootstrap starting.
I20260929 11:37:52.593181 24453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:52.594444 24453 log.cc:824] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:37:52 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:52.597550 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: No bootstrap required, opened a new log
I20260929 11:37:52.605557 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:52.583105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.6.9.254:44123)
0929 11:37:52.583827 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:52.583839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:52.583883 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:37:52.585594 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:52.585617 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:52.586032 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:52.586347 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:52.588116 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:52.589437 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:52.591207 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:52.591867 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:52.591999 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:52.596979 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:52.596990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:52.597008 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:52.597057 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:52.600972 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:52.601674 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:52.601683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:52.601696 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:52.601927 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:52.602246 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:52.602251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:52.602255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:52.602987 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:52.603286 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:52.603646 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 11:37:52.604141 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":330,"threads_started":1}
I20260929 11:37:52.607563 24453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } }
I20260929 11:37:52.607919 24453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:52.608026 24453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f1e3fe286491dad52eb6dc9d3a4c5, State: Initialized, Role: FOLLOWER
I20260929 11:37:52.608606 24453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } }
I20260929 11:37:52.608912 24453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:52.609057 24453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:52.609206 24453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:52.611644 24453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } }
I20260929 11:37:52.612166 24453 leader_election.cc:302] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 749f1e3fe286491dad52eb6dc9d3a4c5; no voters:
I20260929 11:37:52.612742 24453 leader_election.cc:288] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:37:52.612879 24458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:52.613519 24458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [term 1 LEADER]: Becoming Leader. State: Replica: 749f1e3fe286491dad52eb6dc9d3a4c5, State: Running, Role: LEADER
I20260929 11:37:52.613991 24458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } }
I20260929 11:37:52.614684 24453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:52.616678 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } } }
I20260929 11:37:52.617051 24460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 749f1e3fe286491dad52eb6dc9d3a4c5. Latest consensus state: current_term: 1 leader_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "749f1e3fe286491dad52eb6dc9d3a4c5" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 44123 } } }
I20260929 11:37:52.617234 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:52.617352 24460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:52.618633 24467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:52.623697 24467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:52.633467 24467 catalog_manager.cc:1347] Generated new cluster ID: dfe8eeebc3f34a2b98cde64a7edced24
I20260929 11:37:52.633597 24467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:52.653568 24467 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:37:52.655547 24467 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:52.665987 24467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: Generated new TSK 0
I20260929 11:37:52.667088 24467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:37:52.731958 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:53.058168 24479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:53.058454 24479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:53.058578 24479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:53.065845 24479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:53.065987 24479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:53.066143 24479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:53.076128 24479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:53.077581 24479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:53.078950 24479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:53.092193 24484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:53.092200 24487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:53.092417 24485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:53.093223 24479 server_base.cc:1034] running on GCE node
I20260929 11:37:53.093828 24479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:53.094806 24479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:53.096004 24479 hybrid_clock.cc:648] HybridClock initialized: now 1790681873095952 us; error 43 us; skew 500 ppm
I20260929 11:37:53.096309 24479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:53 dist-test-slave-c0l9 krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:53.099566 24479 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:37:53.101245 24479 webserver.cc:460] Webserver started at http://127.6.9.193:45663/ using document root <none> and password file <none>
I20260929 11:37:53.101971 24479 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:53.102094 24479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:53.102403 24479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:53.105417 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "b3a94eea2f9a4ee1882fde9ce6c56216"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "be09ccdb5118440eadc302932701792d"
server_key_iv: "5fa158d86dc6cbe53d87ec6d20e293cf"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.106238 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "b3a94eea2f9a4ee1882fde9ce6c56216"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "be09ccdb5118440eadc302932701792d"
server_key_iv: "5fa158d86dc6cbe53d87ec6d20e293cf"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.111426 24479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:37:53.115043 24494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.116539 24479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:37:53.116745 24479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "b3a94eea2f9a4ee1882fde9ce6c56216"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "be09ccdb5118440eadc302932701792d"
server_key_iv: "5fa158d86dc6cbe53d87ec6d20e293cf"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.116954 24479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:53.136798 24479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:53.141170 24479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:53.141469 24479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:53.142438 24479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:53.143738 24479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:53.143814 24479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.143937 24479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:53.143990 24479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.173187 24479 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:43485
I20260929 11:37:53.173238 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:43485 every 8 connection(s)
I20260929 11:37:53.174443 24479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:53.178736 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24479
I20260929 11:37:53.178915 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:37:53.179352 6183 external_mini_cluster.cc:1428] Setting key 9423e6f17b326e2487e928b90d2b5307
Sep 29 11:37:53 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:53.194898 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:53.176896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:35929 (local address 127.6.9.254:44123)
0929 11:37:53.177345 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:53.177349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:53.178210 (+ 861us) server_negotiation.cc:407] Connection header received
0929 11:37:53.180216 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:53.180221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:53.180306 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:53.180419 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:53.182053 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.183092 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.184582 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.185125 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.185246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:53.188737 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:53.188744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:53.188749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:53.188791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:53.191691 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:53.192555 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:53.192561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:53.192564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:53.192739 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:53.193138 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:53.193143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:53.193147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:53.193472 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:53.193601 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:53.194304 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 11:37:53.194542 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260929 11:37:53.195815 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:53.177331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44123 (local address 127.6.9.193:35929)
0929 11:37:53.178046 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:53.178086 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:53.179929 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:53.180685 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:53.180697 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:53.181295 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:53.181881 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:53.181898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.183219 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.183222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:53.184452 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:53.184463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.185323 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.185328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:53.185542 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:53.186182 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:53.186209 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:53.188461 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:53.191856 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:53.191866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:53.191885 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:53.192426 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:53.192834 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:53.192837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:53.192840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:53.193038 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:53.193577 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:53.193583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:53.193937 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 11:37:53.194176 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 11:37:53.194601 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":269,"threads_started":1}
I20260929 11:37:53.197392 24607 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44123
I20260929 11:37:53.197733 24607 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:53.198621 24607 heartbeater.cc:511] Master 127.6.9.254:44123 requested a full tablet report, sending...
I20260929 11:37:53.201071 24418 ts_manager.cc:194] Registered new tserver with Master: b3a94eea2f9a4ee1882fde9ce6c56216 (127.6.9.193:43485)
I20260929 11:37:53.202610 24418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:35929
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:37:53.233423 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:53.563920 24613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:53.564209 24613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:53.564303 24613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:53.571626 24613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:53.571759 24613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:53.571914 24613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:53.581929 24613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:53.583217 24613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:53.584494 24613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:53.597846 24618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:53.597826 24619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:53.597776 24621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:53.598924 24613 server_base.cc:1034] running on GCE node
I20260929 11:37:53.599534 24613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:53.600490 24613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:53.601785 24613 hybrid_clock.cc:648] HybridClock initialized: now 1790681873601713 us; error 61 us; skew 500 ppm
I20260929 11:37:53.602101 24613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:53 dist-test-slave-c0l9 krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:53.605569 24613 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:37:53.607066 24613 webserver.cc:460] Webserver started at http://127.6.9.194:33435/ using document root <none> and password file <none>
I20260929 11:37:53.607738 24613 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:53.607836 24613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:53.608083 24613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:53.611016 24613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "dd52af399b4f40789933c6cf91036041"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "ef28da33417b73416278b9fd9cb85141"
server_key_iv: "da30e0dfef965e35855e4a95c0ed07fd"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.611788 24613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "dd52af399b4f40789933c6cf91036041"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "ef28da33417b73416278b9fd9cb85141"
server_key_iv: "da30e0dfef965e35855e4a95c0ed07fd"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.617214 24613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 11:37:53.621116 24628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.622632 24613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:37:53.622808 24613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "dd52af399b4f40789933c6cf91036041"
format_stamp: "Formatted at 2026-09-29 11:37:53 on dist-test-slave-c0l9"
server_key: "ef28da33417b73416278b9fd9cb85141"
server_key_iv: "da30e0dfef965e35855e4a95c0ed07fd"
server_key_version: "encryptionkey@0"
I20260929 11:37:53.623006 24613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:53.648932 24613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:53.653538 24613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:53.653832 24613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:53.654847 24613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:53.656299 24613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:53.656381 24613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.656507 24613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:53.656559 24613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:53.686492 24613 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:42643
I20260929 11:37:53.686537 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:42643 every 8 connection(s)
I20260929 11:37:53.687768 24613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:53.697904 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24613
I20260929 11:37:53.698120 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:37:53.698552 6183 external_mini_cluster.cc:1428] Setting key c502f0196b51596b485293d7b6927b6b
Sep 29 11:37:53 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681873, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:53.708406 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:53.690396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:53495 (local address 127.6.9.254:44123)
0929 11:37:53.690629 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:53.690636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:53.691786 (+ 1150us) server_negotiation.cc:407] Connection header received
0929 11:37:53.693699 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:53.693704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:53.693782 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:53.693884 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:53.695403 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.696172 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.698097 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.698584 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.698720 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:53.702892 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:53.702897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:53.702901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:53.702932 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:53.705414 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:53.706255 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:53.706261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:53.706264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:53.706394 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:53.706815 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:53.706818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:53.706820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:53.707062 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:53.707165 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:53.707973 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 11:37:53.708150 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:37:53.709913 24743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:53.690882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44123 (local address 127.6.9.194:53495)
0929 11:37:53.691606 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:53.691652 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:53.693419 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:53.694087 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:53.694096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:53.694622 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:53.695212 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:53.695230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.696324 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.696327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:53.697942 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:53.697958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:53.698815 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:53.698823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:53.699086 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:53.699853 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:53.699884 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:53.702647 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:53.705549 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:53.705558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:53.705577 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:53.706139 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:53.706487 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:53.706491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:53.706494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:53.706718 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:53.707163 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:53.707170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:53.707582 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 11:37:53.707880 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 11:37:53.708416 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260929 11:37:53.711506 24741 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44123
I20260929 11:37:53.711807 24741 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:53.712487 24741 heartbeater.cc:511] Master 127.6.9.254:44123 requested a full tablet report, sending...
I20260929 11:37:53.713929 24418 ts_manager.cc:194] Registered new tserver with Master: dd52af399b4f40789933c6cf91036041 (127.6.9.194:42643)
I20260929 11:37:53.714759 24418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:53495
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:37:53.751353 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:54.078373 24746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:54.078701 24746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:54.078840 24746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:54.086242 24746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:54.086380 24746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:54.086539 24746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:54.096469 24746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:54.097865 24746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:54.099210 24746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:54.112203 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:54.112301 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:54.112319 24751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:54.112866 24746 server_base.cc:1034] running on GCE node
I20260929 11:37:54.113575 24746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:54.114562 24746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:54.115751 24746 hybrid_clock.cc:648] HybridClock initialized: now 1790681874115705 us; error 43 us; skew 500 ppm
I20260929 11:37:54.116050 24746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:54 dist-test-slave-c0l9 krb5kdc[24373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:54.119025 24746 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:37:54.120477 24746 webserver.cc:460] Webserver started at http://127.6.9.195:35057/ using document root <none> and password file <none>
I20260929 11:37:54.121349 24746 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:54.121485 24746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:54.121788 24746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:54.124758 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "9165463dae0143d0b8bc9161b50126ff"
format_stamp: "Formatted at 2026-09-29 11:37:54 on dist-test-slave-c0l9"
server_key: "c8dc9c494263d50b0437800806e4e98b"
server_key_iv: "357774786c2efd12bbe5f7c1ee67ad6b"
server_key_version: "encryptionkey@0"
I20260929 11:37:54.125540 24746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "9165463dae0143d0b8bc9161b50126ff"
format_stamp: "Formatted at 2026-09-29 11:37:54 on dist-test-slave-c0l9"
server_key: "c8dc9c494263d50b0437800806e4e98b"
server_key_iv: "357774786c2efd12bbe5f7c1ee67ad6b"
server_key_version: "encryptionkey@0"
I20260929 11:37:54.130682 24746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:37:54.134609 24761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:54.136309 24746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 11:37:54.136507 24746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "9165463dae0143d0b8bc9161b50126ff"
format_stamp: "Formatted at 2026-09-29 11:37:54 on dist-test-slave-c0l9"
server_key: "c8dc9c494263d50b0437800806e4e98b"
server_key_iv: "357774786c2efd12bbe5f7c1ee67ad6b"
server_key_version: "encryptionkey@0"
I20260929 11:37:54.136734 24746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:54.158952 24746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:54.163929 24746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:54.164252 24746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:54.165588 24746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:54.167152 24746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:54.167266 24746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:54.167383 24746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:54.167450 24746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:54.198766 24746 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:40131
I20260929 11:37:54.198825 24873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:40131 every 8 connection(s)
I20260929 11:37:54.200037 24746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:54.205668 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24746
I20260929 11:37:54.205833 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:37:54.206095 24607 heartbeater.cc:503] Master 127.6.9.254:44123 was elected leader, sending a full tablet report...
I20260929 11:37:54.206391 6183 external_mini_cluster.cc:1428] Setting key e2f6b6636849ff212e1daa222ccec3a1
Sep 29 11:37:54 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681874, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:54.219513 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.202607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:51929 (local address 127.6.9.254:44123)
0929 11:37:54.202821 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.202827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.203854 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 11:37:54.205681 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.205687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.205777 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.205883 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:54.207778 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.208645 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.210226 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.210630 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.210743 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.213954 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:54.213959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:54.213962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:54.213993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:54.216398 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.217136 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.217141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.217143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.217275 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.217705 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.217709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.217711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.218067 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:54.218208 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.218913 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.219148 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:37:54.220620 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.203016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:44123 (local address 127.6.9.195:51929)
0929 11:37:54.203697 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.203734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.205421 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.206164 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.206175 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.206887 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:54.207605 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.207633 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.208809 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.208813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.210083 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.210095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.210844 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.210851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.211065 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.211652 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:37:54.211675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:54.213756 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:54.216524 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:54.216534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:54.216551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:54.217024 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:54.217396 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:37:54.217401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:37:54.217405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:37:54.217593 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:37:54.218182 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:54.218189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:54.218529 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.218771 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.219374 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260929 11:37:54.222107 24874 heartbeater.cc:347] Connected to a master server at 127.6.9.254:44123
I20260929 11:37:54.222430 24874 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:54.223147 24874 heartbeater.cc:511] Master 127.6.9.254:44123 requested a full tablet report, sending...
I20260929 11:37:54.224648 24418 ts_manager.cc:194] Registered new tserver with Master: 9165463dae0143d0b8bc9161b50126ff (127.6.9.195:40131)
I20260929 11:37:54.225535 24418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:51929
I20260929 11:37:54.234264 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:37:54.251843 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.238282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.6.9.254:44123)
0929 11:37:54.238469 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.238477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.238899 (+ 422us) server_negotiation.cc:407] Connection header received
0929 11:37:54.239015 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.239019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.239082 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.239159 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:54.240621 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.241437 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.242712 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.243133 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.243222 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.245862 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:54.245867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:54.245871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:54.245905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:54.248355 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.249257 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.249263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.249266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.249476 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.250091 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.250097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.250100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.250538 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:54.250818 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.251298 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.251496 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:37:54.256301 24418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:37:54.263257 24418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:37:54.294210 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.275399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:42643 (local address 127.0.0.1:33436)
0929 11:37:54.285029 (+ 9630us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.285088 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.285293 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.286429 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.286437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.286482 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:54.286946 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.286958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.289425 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.289430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.291689 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.291700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.293253 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.293259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.293456 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.293509 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.293637 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.293830 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7843,"thread_start_us":9516,"threads_started":1}
I20260929 11:37:54.295228 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.275535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43485 (local address 127.0.0.1:54908)
0929 11:37:54.286280 (+ 10745us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.286300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.286433 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.288378 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.288382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.288405 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:54.288874 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.288890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.290554 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.290558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.292320 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.292335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.294580 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.294585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.294753 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.294787 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.294841 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.294936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10604,"mutex_wait_us":2,"thread_start_us":10984,"threads_started":1}
I20260929 11:37:54.295269 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.275517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.6.9.194:42643)
0929 11:37:54.283039 (+ 7522us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.283052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.285497 (+ 2445us) server_negotiation.cc:407] Connection header received
0929 11:37:54.285605 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.285613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.285909 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.286105 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:54.287131 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.289293 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.291844 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.293041 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.293167 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.293367 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.294777 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.294961 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7308,"thread_start_us":7175,"threads_started":1}
I20260929 11:37:54.295915 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.287240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:40131 (local address 127.0.0.1:49858)
0929 11:37:54.288039 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.288054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.288236 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.289124 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.289128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.289150 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:54.289527 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.289539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.291432 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.291437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.293209 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.293225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.295207 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.295213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.295376 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.295407 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.295489 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.295608 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":540,"threads_started":1}
I20260929 11:37:54.296304 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.276109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.6.9.193:43485)
0929 11:37:54.287764 (+ 11655us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.287772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.287811 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:37:54.287908 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.287915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.288161 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.289085 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:54.289317 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.290418 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.292486 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.294353 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.295541 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.295766 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.295830 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.295997 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11499,"thread_start_us":11322,"threads_started":1}
I20260929 11:37:54.296738 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.286169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.6.9.195:40131)
0929 11:37:54.287229 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.287239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.288142 (+ 903us) server_negotiation.cc:407] Connection header received
0929 11:37:54.288423 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.288431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.288636 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.288823 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:54.289713 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.291282 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.293387 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.294704 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.294830 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.295059 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.296236 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.296405 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":691,"threads_started":1}
I20260929 11:37:54.299207 24676 tablet_service.cc:1467] Processing CreateTablet for tablet 6659065cec40464ca8eb395931f9cf95 (DEFAULT_TABLE table=test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:54.299484 24809 tablet_service.cc:1467] Processing CreateTablet for tablet 6659065cec40464ca8eb395931f9cf95 (DEFAULT_TABLE table=test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:54.299765 24542 tablet_service.cc:1467] Processing CreateTablet for tablet 6659065cec40464ca8eb395931f9cf95 (DEFAULT_TABLE table=test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:54.300848 24676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6659065cec40464ca8eb395931f9cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:54.300853 24809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6659065cec40464ca8eb395931f9cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:54.301316 24542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6659065cec40464ca8eb395931f9cf95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:54.311766 24892 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Bootstrap starting.
I20260929 11:37:54.315056 24892 tablet_bootstrap.cc:654] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:54.315387 24893 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Bootstrap starting.
I20260929 11:37:54.316180 24894 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Bootstrap starting.
I20260929 11:37:54.316843 24892 log.cc:824] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:54.319654 24893 tablet_bootstrap.cc:654] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:54.319706 24894 tablet_bootstrap.cc:654] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:54.320456 24892 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: No bootstrap required, opened a new log
I20260929 11:37:54.320765 24893 log.cc:824] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:54.320801 24892 ts_tablet_manager.cc:1397] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:37:54.320995 24894 log.cc:824] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:54.323439 24893 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: No bootstrap required, opened a new log
I20260929 11:37:54.323676 24893 ts_tablet_manager.cc:1397] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 11:37:54.329041 24894 tablet_bootstrap.cc:492] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: No bootstrap required, opened a new log
I20260929 11:37:54.329371 24894 ts_tablet_manager.cc:1397] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20260929 11:37:54.330725 24892 raft_consensus.cc:348] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.331113 24892 raft_consensus.cc:374] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:54.331238 24892 raft_consensus.cc:729] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9165463dae0143d0b8bc9161b50126ff, State: Initialized, Role: FOLLOWER
I20260929 11:37:54.331844 24892 consensus_queue.cc:260] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.332922 24874 heartbeater.cc:503] Master 127.6.9.254:44123 was elected leader, sending a full tablet report...
I20260929 11:37:54.333130 24893 raft_consensus.cc:348] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.333586 24893 raft_consensus.cc:374] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:54.333747 24893 raft_consensus.cc:729] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a94eea2f9a4ee1882fde9ce6c56216, State: Initialized, Role: FOLLOWER
I20260929 11:37:54.334555 24892 ts_tablet_manager.cc:1428] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 11:37:54.334481 24893 consensus_queue.cc:260] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.336387 24893 ts_tablet_manager.cc:1428] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 11:37:54.339218 24894 raft_consensus.cc:348] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.339633 24894 raft_consensus.cc:374] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:54.339726 24894 raft_consensus.cc:729] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd52af399b4f40789933c6cf91036041, State: Initialized, Role: FOLLOWER
I20260929 11:37:54.340265 24894 consensus_queue.cc:260] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.341464 24741 heartbeater.cc:503] Master 127.6.9.254:44123 was elected leader, sending a full tablet report...
I20260929 11:37:54.342199 24894 ts_tablet_manager.cc:1428] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 11:37:54.380040 24900 raft_consensus.cc:482] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:54.380378 24900 raft_consensus.cc:504] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.382246 24900 leader_election.cc:288] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3a94eea2f9a4ee1882fde9ce6c56216 (127.6.9.193:43485), 9165463dae0143d0b8bc9161b50126ff (127.6.9.195:40131)
I20260929 11:37:54.389813 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.382841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:32993 (local address 127.6.9.195:40131)
0929 11:37:54.383275 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.383282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.383911 (+ 629us) server_negotiation.cc:407] Connection header received
0929 11:37:54.384126 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.384131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.384211 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.384333 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:54.385163 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.386107 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.387971 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.388805 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.388914 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.389027 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.389445 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.389577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260929 11:37:54.389808 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.382744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:54651 (local address 127.6.9.193:43485)
0929 11:37:54.382967 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.382974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.383800 (+ 826us) server_negotiation.cc:407] Connection header received
0929 11:37:54.384001 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.384008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.384097 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.384180 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:54.385055 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.385797 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.387956 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.388818 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.388967 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.389069 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.389434 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.389567 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:37:54.389997 24902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.382579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43485 (local address 127.6.9.194:54651)
0929 11:37:54.383631 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.383652 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.383831 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.384242 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.384247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.384273 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:54.384868 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.384880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.385953 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.385958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.387761 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.387775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.389068 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.389074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.389261 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.389297 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.389375 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.389517 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":30,"thread_start_us":862,"threads_started":1}
I20260929 11:37:54.390029 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.382692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:40131 (local address 127.6.9.194:32993)
0929 11:37:54.383764 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:54.383793 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:54.383945 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:54.384365 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:54.384368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:54.384406 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:54.384969 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.384982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.386274 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.386279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.387778 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:54.387791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.389033 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.389040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:54.389265 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.389305 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:37:54.389395 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:37:54.389522 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":47,"thread_start_us":506,"threads_started":1}
I20260929 11:37:54.391062 24562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6659065cec40464ca8eb395931f9cf95" candidate_uuid: "dd52af399b4f40789933c6cf91036041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" is_pre_election: true
I20260929 11:37:54.391060 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6659065cec40464ca8eb395931f9cf95" candidate_uuid: "dd52af399b4f40789933c6cf91036041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9165463dae0143d0b8bc9161b50126ff" is_pre_election: true
I20260929 11:37:54.391561 24829 raft_consensus.cc:2454] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd52af399b4f40789933c6cf91036041 in term 0.
I20260929 11:37:54.391561 24562 raft_consensus.cc:2454] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd52af399b4f40789933c6cf91036041 in term 0.
I20260929 11:37:54.392236 24629 leader_election.cc:302] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9165463dae0143d0b8bc9161b50126ff, dd52af399b4f40789933c6cf91036041; no voters:
I20260929 11:37:54.392632 24900 raft_consensus.cc:2781] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:37:54.392807 24900 raft_consensus.cc:482] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:37:54.392915 24900 raft_consensus.cc:3037] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:54.395076 24900 raft_consensus.cc:504] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.395694 24900 leader_election.cc:288] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [CANDIDATE]: Term 1 election: Requested vote from peers b3a94eea2f9a4ee1882fde9ce6c56216 (127.6.9.193:43485), 9165463dae0143d0b8bc9161b50126ff (127.6.9.195:40131)
I20260929 11:37:54.396230 24562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6659065cec40464ca8eb395931f9cf95" candidate_uuid: "dd52af399b4f40789933c6cf91036041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216"
I20260929 11:37:54.396270 24829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6659065cec40464ca8eb395931f9cf95" candidate_uuid: "dd52af399b4f40789933c6cf91036041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9165463dae0143d0b8bc9161b50126ff"
I20260929 11:37:54.396474 24829 raft_consensus.cc:3037] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:54.396474 24562 raft_consensus.cc:3037] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:54.399734 24829 raft_consensus.cc:2454] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd52af399b4f40789933c6cf91036041 in term 1.
I20260929 11:37:54.399734 24562 raft_consensus.cc:2454] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd52af399b4f40789933c6cf91036041 in term 1.
I20260929 11:37:54.400239 24631 leader_election.cc:302] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a94eea2f9a4ee1882fde9ce6c56216, dd52af399b4f40789933c6cf91036041; no voters:
I20260929 11:37:54.400575 24900 raft_consensus.cc:2781] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:54.401039 24900 raft_consensus.cc:686] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 1 LEADER]: Becoming Leader. State: Replica: dd52af399b4f40789933c6cf91036041, State: Running, Role: LEADER
I20260929 11:37:54.401508 24900 consensus_queue.cc:237] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } }
I20260929 11:37:54.406399 24418 catalog_manager.cc:5495] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd52af399b4f40789933c6cf91036041 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "dd52af399b4f40789933c6cf91036041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd52af399b4f40789933c6cf91036041" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 42643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 } health_report { overall_health: UNKNOWN } } }
W20260929 11:37:54.426826 24608 tablet.cc:2369] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:37:54.439767 24742 tablet.cc:2369] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:54.446270 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.440746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.6.9.194:42643)
0929 11:37:54.440945 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.440951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.441019 (+ 68us) server_negotiation.cc:407] Connection header received
0929 11:37:54.441244 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.441250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.441336 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.441492 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:54.441913 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.442631 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.444048 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.444454 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.444560 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.445178 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:54.445669 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:37:54.445841 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.445969 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.446016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 11:37:54.451608 24875 tablet.cc:2369] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:54.454819 24829 raft_consensus.cc:1261] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 1 FOLLOWER]: Refusing update from remote peer dd52af399b4f40789933c6cf91036041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:54.454819 24562 raft_consensus.cc:1261] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 1 FOLLOWER]: Refusing update from remote peer dd52af399b4f40789933c6cf91036041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:54.455971 24903 consensus_queue.cc:1035] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9165463dae0143d0b8bc9161b50126ff" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:54.456326 24900 consensus_queue.cc:1035] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a94eea2f9a4ee1882fde9ce6c56216" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:54.466838 24907 mvcc.cc:205] Tried to move back new op lower bound from 7334632957756231680 to 7334632957552922624. Current Snapshot: MvccSnapshot[applied={T|T < 7334632957756231680}]
I20260929 11:37:54.480569 24418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:54.481038 24418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:54.485411 24418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 749f1e3fe286491dad52eb6dc9d3a4c5: Sending DeleteTablet for 3 replicas of tablet 6659065cec40464ca8eb395931f9cf95
I20260929 11:37:54.486698 24809 tablet_service.cc:1514] Processing DeleteTablet for tablet 6659065cec40464ca8eb395931f9cf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:54 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:49858
I20260929 11:37:54.486716 24542 tablet_service.cc:1514] Processing DeleteTablet for tablet 6659065cec40464ca8eb395931f9cf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:54 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:54908
I20260929 11:37:54.486719 24676 tablet_service.cc:1514] Processing DeleteTablet for tablet 6659065cec40464ca8eb395931f9cf95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:54 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:33436
I20260929 11:37:54.494128 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24479
I20260929 11:37:54.498272 24918 tablet_replica.cc:330] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: stopping tablet replica
I20260929 11:37:54.498976 24918 raft_consensus.cc:2229] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:54.499666 24918 raft_consensus.cc:2258] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:54.500564 24917 tablet_replica.cc:330] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: stopping tablet replica
I20260929 11:37:54.501116 24917 raft_consensus.cc:2229] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:54.501744 24917 raft_consensus.cc:2258] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:54.503252 24918 ts_tablet_manager.cc:1905] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:54.503885 24917 ts_tablet_manager.cc:1905] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:54.510560 24918 ts_tablet_manager.cc:1918] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:54.511023 24917 ts_tablet_manager.cc:1918] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:54.511270 24917 log.cc:1196] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/6659065cec40464ca8eb395931f9cf95
I20260929 11:37:54.511904 24917 ts_tablet_manager.cc:1939] T 6659065cec40464ca8eb395931f9cf95 P b3a94eea2f9a4ee1882fde9ce6c56216: Deleting consensus metadata
I20260929 11:37:54.513937 24404 catalog_manager.cc:4841] TS b3a94eea2f9a4ee1882fde9ce6c56216 (127.6.9.193:43485): tablet 6659065cec40464ca8eb395931f9cf95 (table test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]) successfully deleted
I20260929 11:37:54.514366 24918 log.cc:1196] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/6659065cec40464ca8eb395931f9cf95
I20260929 11:37:54.514909 24918 ts_tablet_manager.cc:1939] T 6659065cec40464ca8eb395931f9cf95 P 9165463dae0143d0b8bc9161b50126ff: Deleting consensus metadata
I20260929 11:37:54.515470 24916 tablet_replica.cc:330] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: stopping tablet replica
I20260929 11:37:54.516841 24402 catalog_manager.cc:4841] TS 9165463dae0143d0b8bc9161b50126ff (127.6.9.195:40131): tablet 6659065cec40464ca8eb395931f9cf95 (table test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]) successfully deleted
I20260929 11:37:54.517936 24916 raft_consensus.cc:2229] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 1 LEADER]: Raft consensus shutting down.
Sep 29 11:37:54 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:37:54.518584 24916 raft_consensus.cc:2258] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:54.520556 24916 ts_tablet_manager.cc:1905] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:54.526662 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.496511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.6.9.193:43485)
0929 11:37:54.496755 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.496763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.508033 (+ 11270us) server_negotiation.cc:407] Connection header received
0929 11:37:54.508567 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.508573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.508704 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.508786 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:54.510641 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.511735 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.513625 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.514175 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.514303 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.519410 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:54.519427 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:54.519445 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:54.519496 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:54.523330 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.524214 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.524225 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.524228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.524410 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.524871 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.524876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.524879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.525356 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:54.525660 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.526000 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.526243 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:37:54.526721 24916 ts_tablet_manager.cc:1918] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:54.526926 24916 log.cc:1196] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/6659065cec40464ca8eb395931f9cf95
I20260929 11:37:54.527364 24602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:54.527573 24916 ts_tablet_manager.cc:1939] T 6659065cec40464ca8eb395931f9cf95 P dd52af399b4f40789933c6cf91036041: Deleting consensus metadata
I20260929 11:37:54.530356 24402 catalog_manager.cc:4841] TS dd52af399b4f40789933c6cf91036041 (127.6.9.194:42643): tablet 6659065cec40464ca8eb395931f9cf95 (table test-table [id=607b04440f1d4dc0ab8d8b1b3ba28dff]) successfully deleted
I20260929 11:37:54.648381 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24479
I20260929 11:37:54.670135 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24613
Sep 29 11:37:54 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:37:54.688474 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.670989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.6.9.194:42643)
0929 11:37:54.671179 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.671186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.671215 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:37:54.671283 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.671288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.671387 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.671499 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:54.672994 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.674168 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.675500 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.676190 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.676324 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.680620 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:54.680640 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:54.680687 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:54.680738 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:54.684927 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.685895 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.685905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.685907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.686174 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.686629 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.686636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.686639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.687099 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:54.687438 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.687770 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.688015 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:37:54.689307 24736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:54.806464 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24613
I20260929 11:37:54.822687 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24746
Sep 29 11:37:54 dist-test-slave-c0l9 krb5kdc[24373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:37:54.838635 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:54.823471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.6.9.195:40131)
0929 11:37:54.823649 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:54.823655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:54.823696 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:37:54.823906 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:54.823910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:54.823988 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:54.824075 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:54.825495 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.826235 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.827538 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:54.827959 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:54.828073 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:54.832772 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:54.832790 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:54.832807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:54.832869 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:54.835607 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.836456 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.836464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.836468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.836656 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:54.837164 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:54.837170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:54.837173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:54.837565 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:54.837831 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:54.838139 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 11:37:54.838311 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:37:54.839274 24869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:54.929342 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24746
I20260929 11:37:54.943084 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24386
I20260929 11:37:54.944187 24448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:55.054821 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24386
2026-09-29T11:37:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T11:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:37:55Z Disabled control of system clock
I20260929 11:37:55.126322 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43251
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:43251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:37:55.464776 24927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:55.465122 24927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:55.465211 24927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:55.472641 24927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:55.472796 24927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:55.472843 24927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:55.472905 24927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:55.472949 24927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:55.482352 24927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:43251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:43251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:55.483680 24927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:55.485563 24927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:55.497720 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:55.497691 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:55.497655 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:55.498441 24927 server_base.cc:1034] running on GCE node
I20260929 11:37:55.499316 24927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:55.500819 24927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:55.502023 24927 hybrid_clock.cc:648] HybridClock initialized: now 1790681875501977 us; error 46 us; skew 500 ppm
I20260929 11:37:55.502384 24927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:55.504730 24927 webserver.cc:460] Webserver started at http://127.6.9.254:46857/ using document root <none> and password file <none>
I20260929 11:37:55.505420 24927 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:55.505524 24927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:55.505832 24927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:55.508919 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "25b2cac8c5b94de6b43f056b4592ef00"
format_stamp: "Formatted at 2026-09-29 11:37:55 on dist-test-slave-c0l9"
server_key: "8cb3b816190467857544795c9a8c3ceb"
server_key_iv: "68df3a8bfa01510fc31f15b92da518f5"
server_key_version: "encryptionkey@0"
I20260929 11:37:55.509781 24927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "25b2cac8c5b94de6b43f056b4592ef00"
format_stamp: "Formatted at 2026-09-29 11:37:55 on dist-test-slave-c0l9"
server_key: "8cb3b816190467857544795c9a8c3ceb"
server_key_iv: "68df3a8bfa01510fc31f15b92da518f5"
server_key_version: "encryptionkey@0"
I20260929 11:37:55.515664 24927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:37:55.519443 24941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:55.521036 24927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:37:55.521225 24927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "25b2cac8c5b94de6b43f056b4592ef00"
format_stamp: "Formatted at 2026-09-29 11:37:55 on dist-test-slave-c0l9"
server_key: "8cb3b816190467857544795c9a8c3ceb"
server_key_iv: "68df3a8bfa01510fc31f15b92da518f5"
server_key_version: "encryptionkey@0"
I20260929 11:37:55.521479 24927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:55.561092 24927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:55.562012 24927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:55.562296 24927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:55.580369 24927 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:43251
I20260929 11:37:55.580391 24992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:43251 every 8 connection(s)
I20260929 11:37:55.581828 24927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:55.586011 24993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:55.590338 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 24927
I20260929 11:37:55.590562 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:37:55.591140 6183 external_mini_cluster.cc:1428] Setting key a699923c332e4daf5f6e5376b0a616c1
I20260929 11:37:55.600330 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: Bootstrap starting.
I20260929 11:37:55.603782 24993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:55.605020 24993 log.cc:824] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:55.605331 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:55.593437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.6.9.254:43251)
0929 11:37:55.594580 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:55.594593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:55.594639 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:37:55.596348 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:55.596374 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:55.596995 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:55.597405 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:55.598067 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:55.599379 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:55.600882 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:55.601601 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:55.601780 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:55.602940 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:55.602949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:55.602964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:55.603507 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:55.603592 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:55.603613 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:55.603726 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:55.603957 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 11:37:55.604282 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":421,"threads_started":1}
I20260929 11:37:55.607939 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: No bootstrap required, opened a new log
I20260929 11:37:55.615525 24993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } }
I20260929 11:37:55.615933 24993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:55.616015 24993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b2cac8c5b94de6b43f056b4592ef00, State: Initialized, Role: FOLLOWER
I20260929 11:37:55.616583 24993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } }
I20260929 11:37:55.616854 24993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:55.616969 24993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:55.617086 24993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:55.619747 24993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } }
I20260929 11:37:55.620254 24993 leader_election.cc:302] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25b2cac8c5b94de6b43f056b4592ef00; no voters:
I20260929 11:37:55.620901 24993 leader_election.cc:288] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:37:55.621003 24998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:55.621706 24998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [term 1 LEADER]: Becoming Leader. State: Replica: 25b2cac8c5b94de6b43f056b4592ef00, State: Running, Role: LEADER
I20260929 11:37:55.622299 24998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } }
I20260929 11:37:55.623086 24993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:55.625655 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25b2cac8c5b94de6b43f056b4592ef00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } } }
I20260929 11:37:55.626081 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:55.626806 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25b2cac8c5b94de6b43f056b4592ef00. Latest consensus state: current_term: 1 leader_uuid: "25b2cac8c5b94de6b43f056b4592ef00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b2cac8c5b94de6b43f056b4592ef00" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 43251 } } }
I20260929 11:37:55.627095 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00 [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:55.629680 25006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:55.635483 25006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:55.645697 25006 catalog_manager.cc:1347] Generated new cluster ID: 5c815ed2e38243238a9df78d7c39e40c
I20260929 11:37:55.645857 25006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:55.667367 25006 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:37:55.668645 25006 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:55.678622 25006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: Generated new TSK 0
I20260929 11:37:55.679684 25006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:37:55.695701 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--builtin_ntp_servers=127.6.9.212:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:37:56.032125 25017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:56.032411 25017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:56.032498 25017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:56.039794 25017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:56.039911 25017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:56.040017 25017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:37:56.049305 25017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:56.050526 25017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:56.052289 25017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:56.064882 25022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.064901 25023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.064925 25025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:56.065625 25017 server_base.cc:1034] running on GCE node
I20260929 11:37:56.066226 25017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:56.067216 25017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:56.068439 25017 hybrid_clock.cc:648] HybridClock initialized: now 1790681876068376 us; error 52 us; skew 500 ppm
I20260929 11:37:56.068795 25017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:56.071022 25017 webserver.cc:460] Webserver started at http://127.6.9.193:40743/ using document root <none> and password file <none>
I20260929 11:37:56.071727 25017 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:56.071825 25017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:56.072178 25017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:56.075103 25017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "57925e708b1d4739ae616f123698aac7"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "464739c957e897a181c7dccd28244065"
server_key_iv: "db835ffa018f33bfc532bf6ef064db21"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.075922 25017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "57925e708b1d4739ae616f123698aac7"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "464739c957e897a181c7dccd28244065"
server_key_iv: "db835ffa018f33bfc532bf6ef064db21"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.081235 25017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:37:56.084968 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.086462 25017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:56.086649 25017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "57925e708b1d4739ae616f123698aac7"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "464739c957e897a181c7dccd28244065"
server_key_iv: "db835ffa018f33bfc532bf6ef064db21"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.086884 25017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:56.119395 25017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:56.120324 25017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:56.120603 25017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:56.121690 25017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:56.123353 25017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:56.123445 25017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.123554 25017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:56.123626 25017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.154136 25017 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:43399
I20260929 11:37:56.154208 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:43399 every 8 connection(s)
I20260929 11:37:56.155442 25017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:37:56.163972 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25017
I20260929 11:37:56.164136 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:37:56.164598 6183 external_mini_cluster.cc:1428] Setting key 6c6d13e37dc2bd8babedf6e7020e6a4f
I20260929 11:37:56.167515 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--builtin_ntp_servers=127.6.9.212:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:37:56.168574 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:56.157997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:44169 (local address 127.6.9.254:43251)
0929 11:37:56.158461 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:56.158466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:56.159592 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 11:37:56.161731 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:56.161737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:56.161822 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:56.161922 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:56.163101 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.163839 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.165721 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.166157 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.166285 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:56.167810 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:56.167816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:56.167819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:56.167838 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:56.167881 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:56.167893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:56.167960 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:56.168230 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 11:37:56.168316 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260929 11:37:56.170055 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:56.158508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43251 (local address 127.6.9.193:44169)
0929 11:37:56.159420 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:56.159464 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:56.161405 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:56.162202 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:56.162211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:56.162311 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:56.162914 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:56.162931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.163994 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.163998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:56.165573 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:56.165590 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.166403 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.166411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:56.166707 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:56.167427 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:56.167430 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:56.167473 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:56.167562 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:56.167997 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:56.168010 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:56.168019 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:37:56.168227 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 11:37:56.168705 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":280,"threads_started":1}
I20260929 11:37:56.171664 25144 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43251
I20260929 11:37:56.171981 25144 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:56.172936 25144 heartbeater.cc:511] Master 127.6.9.254:43251 requested a full tablet report, sending...
I20260929 11:37:56.175153 24958 ts_manager.cc:194] Registered new tserver with Master: 57925e708b1d4739ae616f123698aac7 (127.6.9.193:43399)
I20260929 11:37:56.176699 24958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.193:44169
W20260929 11:37:56.488025 25148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:56.488317 25148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:56.488410 25148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:56.495935 25148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:56.496049 25148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:56.496158 25148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:37:56.505499 25148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:56.506737 25148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:56.508455 25148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:56.521114 25153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.521184 25154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.521075 25156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:56.526225 25148 server_base.cc:1034] running on GCE node
I20260929 11:37:56.526892 25148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:56.527853 25148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:56.529072 25148 hybrid_clock.cc:648] HybridClock initialized: now 1790681876529013 us; error 47 us; skew 500 ppm
I20260929 11:37:56.529403 25148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:56.531721 25148 webserver.cc:460] Webserver started at http://127.6.9.194:33967/ using document root <none> and password file <none>
I20260929 11:37:56.532373 25148 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:56.532460 25148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:56.532778 25148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:56.535599 25148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "af6efb24fc8441e7a168bb7b1b6fd506"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "17688510860c75a55541ba8fc02b609e"
server_key_iv: "0fefac5384766abc4fb64109dea20dbb"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.536329 25148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "af6efb24fc8441e7a168bb7b1b6fd506"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "17688510860c75a55541ba8fc02b609e"
server_key_iv: "0fefac5384766abc4fb64109dea20dbb"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.541543 25148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 11:37:56.545469 25162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.547037 25148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:56.547237 25148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "af6efb24fc8441e7a168bb7b1b6fd506"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "17688510860c75a55541ba8fc02b609e"
server_key_iv: "0fefac5384766abc4fb64109dea20dbb"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.547487 25148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:56.576339 25148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:56.577396 25148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:56.577667 25148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:56.578722 25148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:56.580365 25148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:56.580466 25148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.580577 25148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:56.580647 25148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:56.612818 25148 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:35705
I20260929 11:37:56.612864 25274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:35705 every 8 connection(s)
I20260929 11:37:56.614121 25148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:37:56.623551 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25148
I20260929 11:37:56.623714 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:37:56.624229 6183 external_mini_cluster.cc:1428] Setting key 3d42af3aac265f8f7f6b90a5ea014ab4
I20260929 11:37:56.627280 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--builtin_ntp_servers=127.6.9.212:45661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:37:56.627292 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:56.616726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:59057 (local address 127.6.9.254:43251)
0929 11:37:56.616896 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:56.616901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:56.618307 (+ 1406us) server_negotiation.cc:407] Connection header received
0929 11:37:56.620095 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:56.620100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:56.620176 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:56.620276 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:56.621514 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.622257 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.624091 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.624709 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.624859 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:56.626485 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:56.626490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:56.626494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:56.626513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:56.626547 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:56.626559 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:56.626634 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:56.626874 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 11:37:56.626980 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:37:56.628427 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:56.617187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43251 (local address 127.6.9.194:59057)
0929 11:37:56.618101 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:56.618144 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:56.619791 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:56.620535 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:56.620545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:56.620685 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:56.621310 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:56.621334 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.622414 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.622418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:56.623925 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:56.623939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:56.624956 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:56.624964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:56.625245 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:56.626088 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:56.626091 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:56.626159 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:56.626290 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:56.626649 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:56.626660 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:56.626669 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:37:56.626852 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 11:37:56.627308 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":280,"threads_started":1}
I20260929 11:37:56.629943 25275 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43251
I20260929 11:37:56.630247 25275 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:56.631049 25275 heartbeater.cc:511] Master 127.6.9.254:43251 requested a full tablet report, sending...
I20260929 11:37:56.632580 24958 ts_manager.cc:194] Registered new tserver with Master: af6efb24fc8441e7a168bb7b1b6fd506 (127.6.9.194:35705)
I20260929 11:37:56.633606 24958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.194:59057
W20260929 11:37:56.953115 25278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:56.953416 25278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:56.953507 25278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:56.960785 25278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:56.960896 25278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:56.961004 25278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:37:56.970355 25278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:43251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:56.971613 25278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:56.973450 25278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:56.986487 25284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.986459 25283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:56.986419 25286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:56.986984 25278 server_base.cc:1034] running on GCE node
I20260929 11:37:56.987671 25278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:56.988683 25278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:56.989930 25278 hybrid_clock.cc:648] HybridClock initialized: now 1790681876989868 us; error 51 us; skew 500 ppm
I20260929 11:37:56.990264 25278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:37:56.992516 25278 webserver.cc:460] Webserver started at http://127.6.9.195:40143/ using document root <none> and password file <none>
I20260929 11:37:56.993220 25278 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:56.993323 25278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:56.993623 25278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:56.996436 25278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "96a45a250d8e454d8d786d9d39564bcf"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "591e58fb4bd839acecd5f71670f3736c"
server_key_iv: "b5a823736b6724b43a27ab00e48e3ecf"
server_key_version: "encryptionkey@0"
I20260929 11:37:56.997211 25278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "96a45a250d8e454d8d786d9d39564bcf"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "591e58fb4bd839acecd5f71670f3736c"
server_key_iv: "b5a823736b6724b43a27ab00e48e3ecf"
server_key_version: "encryptionkey@0"
I20260929 11:37:57.002061 25278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:37:57.005528 25292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:57.006888 25278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:57.007040 25278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "96a45a250d8e454d8d786d9d39564bcf"
format_stamp: "Formatted at 2026-09-29 11:37:56 on dist-test-slave-c0l9"
server_key: "591e58fb4bd839acecd5f71670f3736c"
server_key_iv: "b5a823736b6724b43a27ab00e48e3ecf"
server_key_version: "encryptionkey@0"
I20260929 11:37:57.007232 25278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:57.031756 25278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:57.032648 25278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:57.032987 25278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:57.034013 25278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:37:57.035575 25278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:37:57.035673 25278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:57.035785 25278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:37:57.035861 25278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:57.066010 25278 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:35553
I20260929 11:37:57.066082 25404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:35553 every 8 connection(s)
I20260929 11:37:57.067294 25278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:37:57.071689 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25278
I20260929 11:37:57.071834 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:37:57.072247 6183 external_mini_cluster.cc:1428] Setting key 733472d161f21386c6ffdd3c5ad95946
I20260929 11:37:57.080137 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.069876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:55629 (local address 127.6.9.254:43251)
0929 11:37:57.070081 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.070087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.071569 (+ 1482us) server_negotiation.cc:407] Connection header received
0929 11:37:57.073811 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.073817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.073911 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.073998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:57.075154 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.075895 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.077484 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.077918 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.078029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.079337 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:57.079342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:57.079345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:57.079364 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:57.079397 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:57.079409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:57.079482 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.079695 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.079808 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:37:57.081200 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.070426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:43251 (local address 127.6.9.195:55629)
0929 11:37:57.071373 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.071416 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.073463 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.074266 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.074275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.074379 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:37:57.074991 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.075010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.076034 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.076038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.077348 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.077361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.078141 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.078148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.078376 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.079021 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:37:57.079023 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:37:57.079063 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:37:57.079149 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:37:57.079492 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:37:57.079503 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:37:57.079511 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.079672 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.080098 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
I20260929 11:37:57.082475 25405 heartbeater.cc:347] Connected to a master server at 127.6.9.254:43251
I20260929 11:37:57.082768 25405 heartbeater.cc:464] Registering TS with master...
I20260929 11:37:57.083508 25405 heartbeater.cc:511] Master 127.6.9.254:43251 requested a full tablet report, sending...
I20260929 11:37:57.084954 24958 ts_manager.cc:194] Registered new tserver with Master: 96a45a250d8e454d8d786d9d39564bcf (127.6.9.195:35553)
I20260929 11:37:57.085899 24958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.195:55629
I20260929 11:37:57.087909 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:37:57.098353 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.092329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.6.9.254:43251)
0929 11:37:57.092535 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.092540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.092946 (+ 406us) server_negotiation.cc:407] Connection header received
0929 11:37:57.093113 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.093117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.093196 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.093279 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:57.093823 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.094536 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.095982 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.096424 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.096539 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.097690 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:57.097695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:57.097698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:57.097719 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:57.097773 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:57.097790 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:57.097912 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.098065 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.098120 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:37:57.102464 24958 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:37:57.109774 24958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:37:57.144523 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.125521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.6.9.194:35705)
0929 11:37:57.133926 (+ 8405us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.133938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.135563 (+ 1625us) server_negotiation.cc:407] Connection header received
0929 11:37:57.135922 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.135931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.136195 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.136365 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:57.137714 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.138684 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.141749 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.143071 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.143201 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.143461 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.144021 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.144195 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8239,"thread_start_us":8135,"threads_started":1}
I20260929 11:37:57.144552 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.125238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:35705 (local address 127.0.0.1:55678)
0929 11:37:57.135460 (+ 10222us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.135475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.135655 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.136718 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.136726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.136754 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:57.137502 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.137512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.138848 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.138853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.141527 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.141546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.143581 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.143587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.143778 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.143830 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.143932 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.144159 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10108,"mutex_wait_us":112,"thread_start_us":9514,"threads_started":1}
I20260929 11:37:57.145160 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.122770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:35553 (local address 127.0.0.1:60522)
0929 11:37:57.135678 (+ 12908us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.135695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.135834 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.136603 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.136611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.136642 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:57.137389 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.137411 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.140328 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.140334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.141955 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.141970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.144008 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.144014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.144209 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.144250 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.144319 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.144453 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12098,"mutex_wait_us":85,"spinlock_wait_cycles":22784,"thread_start_us":11874,"threads_started":1}
I20260929 11:37:57.145212 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.123175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.6.9.195:35553)
0929 11:37:57.133960 (+ 10785us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.133970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.135975 (+ 2005us) server_negotiation.cc:407] Connection header received
0929 11:37:57.136086 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.136093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.136319 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.136896 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:57.138848 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.140198 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.142147 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.143516 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.143648 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.143886 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.144699 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.144860 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10576,"thread_start_us":10760,"threads_started":1}
I20260929 11:37:57.144618 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.123428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.6.9.193:43399)
0929 11:37:57.134558 (+ 11130us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.134567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.135427 (+ 860us) server_negotiation.cc:407] Connection header received
0929 11:37:57.135922 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.135930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.136195 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.136365 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:57.137622 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.138629 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.141814 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.143174 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.143312 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.143544 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.144096 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.144282 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10977,"thread_start_us":10871,"threads_started":1}
I20260929 11:37:57.146696 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.123015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57464)
0929 11:37:57.135191 (+ 12176us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.135272 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.135655 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.136653 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.136656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.136727 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:57.137454 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.137468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.138771 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.138776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.141531 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.141546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.143680 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.143687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.143887 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.143920 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.146197 (+ 2277us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.146348 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11883,"mutex_wait_us":144,"spinlock_wait_cycles":7040,"thread_start_us":12418,"threads_started":1}
I20260929 11:37:57.148708 25210 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3237b723ea4711b5c7499992f60660 (DEFAULT_TABLE table=test-table [id=fbc56dfe07d14385a34550a351e80374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:57.148833 25340 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3237b723ea4711b5c7499992f60660 (DEFAULT_TABLE table=test-table [id=fbc56dfe07d14385a34550a351e80374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:57.149995 25210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3237b723ea4711b5c7499992f60660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:57.150259 25340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3237b723ea4711b5c7499992f60660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:57.150279 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 3d3237b723ea4711b5c7499992f60660 (DEFAULT_TABLE table=test-table [id=fbc56dfe07d14385a34550a351e80374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:37:57.151640 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d3237b723ea4711b5c7499992f60660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:37:57.167002 25423 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Bootstrap starting.
I20260929 11:37:57.169865 25425 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Bootstrap starting.
I20260929 11:37:57.170814 25424 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Bootstrap starting.
I20260929 11:37:57.171200 25423 tablet_bootstrap.cc:654] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:57.172564 25425 tablet_bootstrap.cc:654] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:57.172538 25423 log.cc:824] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:57.173869 25425 log.cc:824] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:57.174337 25424 tablet_bootstrap.cc:654] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:57.176410 25424 log.cc:824] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:57.177321 25423 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: No bootstrap required, opened a new log
I20260929 11:37:57.177585 25423 ts_tablet_manager.cc:1397] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 11:37:57.179342 25425 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: No bootstrap required, opened a new log
I20260929 11:37:57.179625 25425 ts_tablet_manager.cc:1397] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 11:37:57.179963 25424 tablet_bootstrap.cc:492] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: No bootstrap required, opened a new log
I20260929 11:37:57.180127 25144 heartbeater.cc:503] Master 127.6.9.254:43251 was elected leader, sending a full tablet report...
I20260929 11:37:57.180245 25424 ts_tablet_manager.cc:1397] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 11:37:57.188310 25423 raft_consensus.cc:348] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.188776 25423 raft_consensus.cc:374] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:57.188897 25423 raft_consensus.cc:729] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6efb24fc8441e7a168bb7b1b6fd506, State: Initialized, Role: FOLLOWER
I20260929 11:37:57.189592 25423 consensus_queue.cc:260] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.190858 25275 heartbeater.cc:503] Master 127.6.9.254:43251 was elected leader, sending a full tablet report...
I20260929 11:37:57.190598 25425 raft_consensus.cc:348] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.191126 25425 raft_consensus.cc:374] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:57.191226 25425 raft_consensus.cc:729] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57925e708b1d4739ae616f123698aac7, State: Initialized, Role: FOLLOWER
I20260929 11:37:57.191082 25424 raft_consensus.cc:348] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.191524 25423 ts_tablet_manager.cc:1428] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260929 11:37:57.191574 25424 raft_consensus.cc:374] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:57.191664 25424 raft_consensus.cc:729] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a45a250d8e454d8d786d9d39564bcf, State: Initialized, Role: FOLLOWER
I20260929 11:37:57.191951 25425 consensus_queue.cc:260] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.192214 25424 consensus_queue.cc:260] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.193993 25425 ts_tablet_manager.cc:1428] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 11:37:57.195907 25405 heartbeater.cc:503] Master 127.6.9.254:43251 was elected leader, sending a full tablet report...
I20260929 11:37:57.196341 25424 ts_tablet_manager.cc:1428] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 11:37:57.268224 25429 raft_consensus.cc:482] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:37:57.268697 25429 raft_consensus.cc:504] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.270823 25429 leader_election.cc:288] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96a45a250d8e454d8d786d9d39564bcf (127.6.9.195:35553), 57925e708b1d4739ae616f123698aac7 (127.6.9.193:43399)
I20260929 11:37:57.277556 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.271128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:54481 (local address 127.6.9.195:35553)
0929 11:37:57.271409 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.271415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.272130 (+ 715us) server_negotiation.cc:407] Connection header received
0929 11:37:57.272322 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.272326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.272393 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.272496 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:57.273188 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.273915 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.275891 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.276690 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.276812 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.276912 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.277213 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.277335 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 11:37:57.277784 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.271458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:44533 (local address 127.6.9.193:43399)
0929 11:37:57.271680 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.271687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.272306 (+ 619us) server_negotiation.cc:407] Connection header received
0929 11:37:57.272456 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.272462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.272537 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.272612 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:37:57.273297 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.273981 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.275888 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.276919 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.277040 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.277133 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.277422 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.277549 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:37:57.277674 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.271167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:35553 (local address 127.6.9.194:54481)
0929 11:37:57.272018 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.272037 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.272187 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.272512 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.272516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.272540 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:57.273028 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.273039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.274038 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.274044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.275690 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.275702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.276872 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.276878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.277029 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.277052 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.277137 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.277270 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":92,"spinlock_wait_cycles":23936,"thread_start_us":499,"threads_started":1}
I20260929 11:37:57.278132 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.271268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.6.9.194:44533)
0929 11:37:57.272199 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.272214 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:37:57.272338 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:37:57.272646 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:37:57.272650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:37:57.272714 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:37:57.273130 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.273141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.274117 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.274122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.275681 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:37:57.275694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.277103 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.277110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:37:57.277268 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.277299 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:37:57.277377 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 11:37:57.277503 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":22,"thread_start_us":594,"threads_started":1}
I20260929 11:37:57.278954 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3237b723ea4711b5c7499992f60660" candidate_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a45a250d8e454d8d786d9d39564bcf" is_pre_election: true
I20260929 11:37:57.279135 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3237b723ea4711b5c7499992f60660" candidate_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57925e708b1d4739ae616f123698aac7" is_pre_election: true
I20260929 11:37:57.279472 25360 raft_consensus.cc:2454] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6efb24fc8441e7a168bb7b1b6fd506 in term 0.
I20260929 11:37:57.279632 25099 raft_consensus.cc:2454] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6efb24fc8441e7a168bb7b1b6fd506 in term 0.
I20260929 11:37:57.280157 25166 leader_election.cc:302] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a45a250d8e454d8d786d9d39564bcf, af6efb24fc8441e7a168bb7b1b6fd506; no voters:
I20260929 11:37:57.280531 25429 raft_consensus.cc:2781] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:37:57.280655 25429 raft_consensus.cc:482] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:37:57.280779 25429 raft_consensus.cc:3037] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:57.282928 25429 raft_consensus.cc:504] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.283507 25429 leader_election.cc:288] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [CANDIDATE]: Term 1 election: Requested vote from peers 96a45a250d8e454d8d786d9d39564bcf (127.6.9.195:35553), 57925e708b1d4739ae616f123698aac7 (127.6.9.193:43399)
I20260929 11:37:57.283993 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3237b723ea4711b5c7499992f60660" candidate_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57925e708b1d4739ae616f123698aac7"
I20260929 11:37:57.283999 25360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d3237b723ea4711b5c7499992f60660" candidate_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a45a250d8e454d8d786d9d39564bcf"
I20260929 11:37:57.284165 25099 raft_consensus.cc:3037] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:57.284174 25360 raft_consensus.cc:3037] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:57.286357 25099 raft_consensus.cc:2454] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6efb24fc8441e7a168bb7b1b6fd506 in term 1.
I20260929 11:37:57.286441 25360 raft_consensus.cc:2454] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6efb24fc8441e7a168bb7b1b6fd506 in term 1.
I20260929 11:37:57.286787 25165 leader_election.cc:302] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57925e708b1d4739ae616f123698aac7, af6efb24fc8441e7a168bb7b1b6fd506; no voters:
I20260929 11:37:57.287113 25429 raft_consensus.cc:2781] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:57.287521 25429 raft_consensus.cc:686] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 1 LEADER]: Becoming Leader. State: Replica: af6efb24fc8441e7a168bb7b1b6fd506, State: Running, Role: LEADER
I20260929 11:37:57.287982 25429 consensus_queue.cc:237] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } }
I20260929 11:37:57.293434 24957 catalog_manager.cc:5495] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 reported cstate change: term changed from 0 to 1, leader changed from <none> to af6efb24fc8441e7a168bb7b1b6fd506 (127.6.9.194). New cstate: current_term: 1 leader_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6efb24fc8441e7a168bb7b1b6fd506" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 35705 } health_report { overall_health: HEALTHY } } }
I20260929 11:37:57.312438 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.307238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.6.9.194:35705)
0929 11:37:57.307416 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.307420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.307438 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:57.307540 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.307544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.307628 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.307754 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:37:57.308137 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.308919 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.310242 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.310674 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.310793 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.311262 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:37:57.311812 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:37:57.311960 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.312112 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.312156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 11:37:57.318930 25406 tablet.cc:2369] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:37:57.321331 25099 raft_consensus.cc:1261] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 1 FOLLOWER]: Refusing update from remote peer af6efb24fc8441e7a168bb7b1b6fd506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:57.321331 25360 raft_consensus.cc:1261] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 1 FOLLOWER]: Refusing update from remote peer af6efb24fc8441e7a168bb7b1b6fd506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:37:57.322433 25429 consensus_queue.cc:1035] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a45a250d8e454d8d786d9d39564bcf" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 35553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:57.322794 25434 consensus_queue.cc:1035] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57925e708b1d4739ae616f123698aac7" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:37:57.334873 25437 mvcc.cc:205] Tried to move back new op lower bound from 7334632969495998464 to 7334632969376845824. Current Snapshot: MvccSnapshot[applied={T|T < 7334632969495998464}]
I20260929 11:37:57.350538 24958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:57.350829 24958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:37:57.354662 24958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25b2cac8c5b94de6b43f056b4592ef00: Sending DeleteTablet for 3 replicas of tablet 3d3237b723ea4711b5c7499992f60660
I20260929 11:37:57.355852 25210 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3237b723ea4711b5c7499992f60660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:57 UTC) from {username='slave'} at 127.0.0.1:55678
I20260929 11:37:57.355885 25340 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3237b723ea4711b5c7499992f60660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:57 UTC) from {username='slave'} at 127.0.0.1:60522
I20260929 11:37:57.357163 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d3237b723ea4711b5c7499992f60660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:37:57 UTC) from {username='slave'} at 127.0.0.1:57464
I20260929 11:37:57.357666 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25017
I20260929 11:37:57.368008 25447 tablet_replica.cc:330] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: stopping tablet replica
I20260929 11:37:57.368721 25447 raft_consensus.cc:2229] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:37:57.369769 25447 raft_consensus.cc:2258] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:57.370900 25446 tablet_replica.cc:330] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: stopping tablet replica
I20260929 11:37:57.371493 25446 raft_consensus.cc:2229] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:57.372005 25448 tablet_replica.cc:330] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: stopping tablet replica
I20260929 11:37:57.372218 25446 raft_consensus.cc:2258] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:57.372483 25448 raft_consensus.cc:2229] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:37:57.373108 25448 raft_consensus.cc:2258] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:37:57.374953 25446 ts_tablet_manager.cc:1905] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:57.375352 25448 ts_tablet_manager.cc:1905] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:57.380986 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.359736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.6.9.193:43399)
0929 11:37:57.359970 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.359977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.371778 (+ 11801us) server_negotiation.cc:407] Connection header received
0929 11:37:57.372343 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.372349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.372480 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.372605 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:57.373382 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.374580 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.376714 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.377383 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.377495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.379006 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:57.379017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:57.379035 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:57.379859 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:57.379928 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:57.379958 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:57.380053 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.380325 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.380455 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:37:57.382301 25139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:57.382826 25446 ts_tablet_manager.cc:1918] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:57.383057 25446 log.cc:1196] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/3d3237b723ea4711b5c7499992f60660
I20260929 11:37:57.383737 25446 ts_tablet_manager.cc:1939] T 3d3237b723ea4711b5c7499992f60660 P 96a45a250d8e454d8d786d9d39564bcf: Deleting consensus metadata
I20260929 11:37:57.385928 24945 catalog_manager.cc:4841] TS 96a45a250d8e454d8d786d9d39564bcf (127.6.9.195:35553): tablet 3d3237b723ea4711b5c7499992f60660 (table test-table [id=fbc56dfe07d14385a34550a351e80374]) successfully deleted
I20260929 11:37:57.385985 25447 ts_tablet_manager.cc:1905] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:37:57.392828 25447 ts_tablet_manager.cc:1918] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:57.393116 25447 log.cc:1196] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/3d3237b723ea4711b5c7499992f60660
I20260929 11:37:57.393589 25447 ts_tablet_manager.cc:1939] T 3d3237b723ea4711b5c7499992f60660 P af6efb24fc8441e7a168bb7b1b6fd506: Deleting consensus metadata
I20260929 11:37:57.395195 24943 catalog_manager.cc:4841] TS af6efb24fc8441e7a168bb7b1b6fd506 (127.6.9.194:35705): tablet 3d3237b723ea4711b5c7499992f60660 (table test-table [id=fbc56dfe07d14385a34550a351e80374]) successfully deleted
I20260929 11:37:57.512389 25448 ts_tablet_manager.cc:1918] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:37:57.512810 25448 log.cc:1196] T 3d3237b723ea4711b5c7499992f60660 P 57925e708b1d4739ae616f123698aac7: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/3d3237b723ea4711b5c7499992f60660
I20260929 11:37:57.513105 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25017
W20260929 11:37:57.527827 24944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.9.193:43399 (error 108)
I20260929 11:37:57.527938 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25148
I20260929 11:37:57.530187 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.529693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57478)
0929 11:37:57.529874 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.529999 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:37:57.530464 24944 catalog_manager.cc:4584] TS 57925e708b1d4739ae616f123698aac7 (127.6.9.193:43399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d3237b723ea4711b5c7499992f60660: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
I20260929 11:37:57.538540 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.528886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.6.9.194:35705)
0929 11:37:57.529079 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.529085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.529202 (+ 117us) server_negotiation.cc:407] Connection header received
0929 11:37:57.529331 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.529337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.529468 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.529597 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:57.530083 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.531037 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.532290 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.532902 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.533021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.537078 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:57.537091 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:57.537107 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:57.537761 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:57.537823 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:57.537844 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:57.537936 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.538087 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.538199 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:37:57.539350 25270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:57.584182 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.583580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57488)
0929 11:37:57.583833 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.583967 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 11:37:57.585373 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.584960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57490)
0929 11:37:57.585111 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.585217 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:37:57.646176 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.645680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57494)
0929 11:37:57.645921 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.646016 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 11:37:57.647202 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.646866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57506)
0929 11:37:57.647015 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.647089 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:37:57.657488 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25148
I20260929 11:37:57.675973 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25278
I20260929 11:37:57.682524 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.676798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.6.9.195:35553)
0929 11:37:57.676948 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:57.676952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:57.676970 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:37:57.677060 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:57.677064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:57.677154 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:57.677242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:57.677568 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.678297 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.679663 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:57.680065 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:57.680165 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:57.681268 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:57.681278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:57.681294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:37:57.681908 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:57.681960 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:37:57.681983 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:57.682092 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:57.682165 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 11:37:57.682263 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:37:57.683185 25400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:57.727661 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.727153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57514)
0929 11:37:57.727380 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.727476 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 11:37:57.728756 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:57.728353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:43399 (local address 127.0.0.1:57524)
0929 11:37:57.728537 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:37:57.728610 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 11:37:57.771651 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25278
I20260929 11:37:57.785331 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 24927
I20260929 11:37:57.786433 24988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:37:57.897717 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 24927
2026-09-29T11:37:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2842 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:37:57 dist-test-slave-c0l9 krb5kdc[25457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:37:57 dist-test-slave-c0l9 krb5kdc[25457](info): set up 2 sockets
Sep 29 11:37:57 dist-test-slave-c0l9 krb5kdc[25457](info): commencing operation
krb5kdc: starting...
W20260929 11:37:58.920871 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:37:59 dist-test-slave-c0l9 krb5kdc[25457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:37:59.162303 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36405
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:36405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:37:59.496145 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:37:59.496454 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:37:59.496544 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:37:59.503836 25470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:37:59.503952 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:37:59.503998 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:37:59.504061 25470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:37:59.504108 25470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:37:59.514070 25470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:36405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:37:59.515362 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:37:59.517038 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:37:59.529161 25476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:59.529209 25475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:37:59.529228 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:37:59.530793 25470 server_base.cc:1034] running on GCE node
I20260929 11:37:59.531585 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:37:59.533200 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:37:59.534442 25470 hybrid_clock.cc:648] HybridClock initialized: now 1790681879534378 us; error 54 us; skew 500 ppm
I20260929 11:37:59.534819 25470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:37:59 dist-test-slave-c0l9 krb5kdc[25457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:37:59.537992 25470 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:37:59.539541 25470 webserver.cc:460] Webserver started at http://127.6.9.254:34119/ using document root <none> and password file <none>
I20260929 11:37:59.540230 25470 fs_manager.cc:359] Metadata directory not provided
I20260929 11:37:59.540349 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:37:59.540650 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:37:59.543774 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "4a3578f4726f46ebbe5e15015f589c7d"
format_stamp: "Formatted at 2026-09-29 11:37:59 on dist-test-slave-c0l9"
server_key: "c509006c48218c3949e22cb4e9507344"
server_key_iv: "9aefb3d3d4371c8632f100fba9206175"
server_key_version: "encryptionkey@0"
I20260929 11:37:59.544524 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "4a3578f4726f46ebbe5e15015f589c7d"
format_stamp: "Formatted at 2026-09-29 11:37:59 on dist-test-slave-c0l9"
server_key: "c509006c48218c3949e22cb4e9507344"
server_key_iv: "9aefb3d3d4371c8632f100fba9206175"
server_key_version: "encryptionkey@0"
I20260929 11:37:59.549819 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:37:59.553256 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:37:59.554623 25470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:37:59.554822 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "4a3578f4726f46ebbe5e15015f589c7d"
format_stamp: "Formatted at 2026-09-29 11:37:59 on dist-test-slave-c0l9"
server_key: "c509006c48218c3949e22cb4e9507344"
server_key_iv: "9aefb3d3d4371c8632f100fba9206175"
server_key_version: "encryptionkey@0"
I20260929 11:37:59.555030 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:37:59.567613 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:37:59.572194 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:37:59.572508 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:37:59.591962 25470 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:36405
I20260929 11:37:59.591984 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:36405 every 8 connection(s)
I20260929 11:37:59.593502 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:37:59.596390 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25470
I20260929 11:37:59.596545 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:37:59.597023 6183 external_mini_cluster.cc:1428] Setting key ef232a46620ba61363c8069ec37a596e
I20260929 11:37:59.598052 25537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:37:59 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:37:59.610680 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: Bootstrap starting.
I20260929 11:37:59.614445 25537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: Neither blocks nor log segments found. Creating new log.
I20260929 11:37:59.615782 25537 log.cc:824] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:37:59.619061 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:37:59.598681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.6.9.254:36405)
0929 11:37:59.599436 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 11:37:59.599452 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:37:59.599783 (+ 331us) server_negotiation.cc:407] Connection header received
0929 11:37:59.601572 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:37:59.601594 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:37:59.601976 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:37:59.602348 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:37:59.603757 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:59.605128 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:59.606390 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:37:59.607063 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:37:59.607251 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:37:59.611373 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:37:59.611382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:37:59.611397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:37:59.611438 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:37:59.614561 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:59.615529 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:59.615537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:59.615543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:59.615724 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:37:59.616200 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:37:59.616204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:37:59.616206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:37:59.617017 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:37:59.617337 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 11:37:59.617699 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:37:59.618054 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":331,"threads_started":1}
I20260929 11:37:59.619138 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: No bootstrap required, opened a new log
I20260929 11:37:59.626870 25537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } }
I20260929 11:37:59.627350 25537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:37:59.627446 25537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3578f4726f46ebbe5e15015f589c7d, State: Initialized, Role: FOLLOWER
I20260929 11:37:59.628012 25537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } }
I20260929 11:37:59.628224 25537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:37:59.628356 25537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:37:59.628505 25537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:37:59.630625 25537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } }
I20260929 11:37:59.631160 25537 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3578f4726f46ebbe5e15015f589c7d; no voters:
I20260929 11:37:59.631784 25537 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:37:59.631906 25542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:37:59.632493 25542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [term 1 LEADER]: Becoming Leader. State: Replica: 4a3578f4726f46ebbe5e15015f589c7d, State: Running, Role: LEADER
I20260929 11:37:59.633118 25542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } }
I20260929 11:37:59.633998 25537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:37:59.635869 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a3578f4726f46ebbe5e15015f589c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } } }
I20260929 11:37:59.636298 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:59.637169 25544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3578f4726f46ebbe5e15015f589c7d. Latest consensus state: current_term: 1 leader_uuid: "4a3578f4726f46ebbe5e15015f589c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3578f4726f46ebbe5e15015f589c7d" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 36405 } } }
I20260929 11:37:59.637565 25544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d [sys.catalog]: This master's current role is: LEADER
I20260929 11:37:59.639590 25551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:37:59.643821 25551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:37:59.653115 25551 catalog_manager.cc:1347] Generated new cluster ID: c04ba22678844ad7a9133e22c414aebb
I20260929 11:37:59.653241 25551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:37:59.671062 25551 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:37:59.672395 25551 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:37:59.683705 25551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: Generated new TSK 0
I20260929 11:37:59.684898 25551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.9.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
Entry for principal kudu/127.6.9.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab.
I20260929 11:37:59.744230 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:00.084453 25563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:00.084795 25563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:00.084941 25563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:00.092213 25563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:00.092337 25563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:00.092485 25563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:38:00.102727 25563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.193.keytab
--principal=kudu/127.6.9.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:00.104108 25563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:00.105501 25563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:00.121994 25569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:00.121978 25571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:00.122248 25568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:00.122875 25563 server_base.cc:1034] running on GCE node
I20260929 11:38:00.123551 25563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:00.124586 25563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:00.125911 25563 hybrid_clock.cc:648] HybridClock initialized: now 1790681880125844 us; error 56 us; skew 500 ppm
I20260929 11:38:00.126243 25563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:00 dist-test-slave-c0l9 krb5kdc[25457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681880, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:00.129385 25563 init.cc:377] Logged in from keytab as kudu/127.6.9.193@KRBTEST.COM (short username kudu)
I20260929 11:38:00.130887 25563 webserver.cc:460] Webserver started at http://127.6.9.193:35853/ using document root <none> and password file <none>
I20260929 11:38:00.131593 25563 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:00.131716 25563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:00.132010 25563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:00.135165 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "1eebb3fe6b0e07ff659e49684205a2c0"
server_key_iv: "e01efba73218da75b6871c2ec16dd29b"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.136019 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "1eebb3fe6b0e07ff659e49684205a2c0"
server_key_iv: "e01efba73218da75b6871c2ec16dd29b"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.141748 25563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:38:00.145606 25578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.147006 25563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:00.147176 25563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "1eebb3fe6b0e07ff659e49684205a2c0"
server_key_iv: "e01efba73218da75b6871c2ec16dd29b"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.147384 25563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:00.170033 25563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:00.174185 25563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:00.174474 25563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:00.175498 25563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:00.176905 25563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:00.176990 25563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.177111 25563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:00.177163 25563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.208226 25563 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:44573
I20260929 11:38:00.208274 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:44573 every 8 connection(s)
I20260929 11:38:00.209537 25563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:38:00.218989 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25563
I20260929 11:38:00.219177 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:38:00.219597 6183 external_mini_cluster.cc:1428] Setting key 34c199d441242dd54fb46342682f88ea
Sep 29 11:38:00 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681880, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.193@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:00.230641 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:00.212037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:35181 (local address 127.6.9.254:36405)
0929 11:38:00.212475 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:00.212480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:00.213394 (+ 914us) server_negotiation.cc:407] Connection header received
0929 11:38:00.215257 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:00.215263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:00.215354 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:00.215452 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:00.217242 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.217991 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.219676 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.220160 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.220302 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:00.224307 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:00.224323 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:00.224328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:00.224372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:00.227490 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:00.228292 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:00.228297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:00.228300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:00.228444 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:00.228921 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:00.228926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:00.228928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:00.229256 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:00.229395 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:00.230086 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 11:38:00.230312 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
I20260929 11:38:00.231761 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:00.212459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36405 (local address 127.6.9.193:35181)
0929 11:38:00.213206 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:00.213249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:00.214965 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:00.215702 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:00.215711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:00.216238 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:00.217037 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:00.217063 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.218158 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.218161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:00.219520 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:00.219534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.220526 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.220539 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:00.220875 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:00.221593 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:00.221618 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:00.224043 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:00.227651 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:00.227658 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:00.227674 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:00.228165 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:00.228551 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:00.228554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:00.228556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:00.228799 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:00.229357 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:00.229363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:00.229704 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 11:38:00.229948 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 11:38:00.230357 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260929 11:38:00.233314 25691 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36405
I20260929 11:38:00.233604 25691 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:00.234349 25691 heartbeater.cc:511] Master 127.6.9.254:36405 requested a full tablet report, sending...
I20260929 11:38:00.236577 25502 ts_manager.cc:194] Registered new tserver with Master: 06a3ff5f39ed4e8d8be0fe6946ee66cf (127.6.9.193:44573)
I20260929 11:38:00.238550 25502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.193@KRBTEST.COM'} at 127.6.9.193:35181
WARNING: no policy specified for kudu/127.6.9.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
Entry for principal kudu/127.6.9.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab.
I20260929 11:38:00.275496 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:00.612437 25697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:00.612849 25697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:00.613016 25697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:00.620771 25697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:00.620925 25697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:00.621114 25697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:38:00.631511 25697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.194.keytab
--principal=kudu/127.6.9.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:00.632958 25697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:00.634320 25697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:00.648335 25703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:00.648350 25705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:00.648566 25702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:00.649228 25697 server_base.cc:1034] running on GCE node
I20260929 11:38:00.649885 25697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:00.650961 25697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:00.652184 25697 hybrid_clock.cc:648] HybridClock initialized: now 1790681880652139 us; error 43 us; skew 500 ppm
I20260929 11:38:00.652487 25697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:00 dist-test-slave-c0l9 krb5kdc[25457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681880, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:00.655735 25697 init.cc:377] Logged in from keytab as kudu/127.6.9.194@KRBTEST.COM (short username kudu)
I20260929 11:38:00.657395 25697 webserver.cc:460] Webserver started at http://127.6.9.194:37423/ using document root <none> and password file <none>
I20260929 11:38:00.658142 25697 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:00.658267 25697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:00.658628 25697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:00.661782 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "043bcda7f23c42b182f998eef7c4577b"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "b3ae4ce29c6c8ebdac4c01454c266c13"
server_key_iv: "9778b4cb2d5334741346d33b68eeb58c"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.662577 25697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "043bcda7f23c42b182f998eef7c4577b"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "b3ae4ce29c6c8ebdac4c01454c266c13"
server_key_iv: "9778b4cb2d5334741346d33b68eeb58c"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.668030 25697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:38:00.671681 25712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.673118 25697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:38:00.673290 25697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "043bcda7f23c42b182f998eef7c4577b"
format_stamp: "Formatted at 2026-09-29 11:38:00 on dist-test-slave-c0l9"
server_key: "b3ae4ce29c6c8ebdac4c01454c266c13"
server_key_iv: "9778b4cb2d5334741346d33b68eeb58c"
server_key_version: "encryptionkey@0"
I20260929 11:38:00.673494 25697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:00.706453 25697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:00.710515 25697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:00.710783 25697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:00.711814 25697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:00.713223 25697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:00.713303 25697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.713436 25697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:00.713490 25697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:00.745038 25697 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:38651
I20260929 11:38:00.745101 25824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:38651 every 8 connection(s)
I20260929 11:38:00.746335 25697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:38:00.750243 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25697
I20260929 11:38:00.750423 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:38:00.750878 6183 external_mini_cluster.cc:1428] Setting key 998466c8b646a49786662b6f660c4639
Sep 29 11:38:00 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681880, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.194@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:00.766438 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:00.748842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:49255 (local address 127.6.9.254:36405)
0929 11:38:00.749278 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:00.749283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:00.750180 (+ 897us) server_negotiation.cc:407] Connection header received
0929 11:38:00.752295 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:00.752300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:00.752368 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:00.752469 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:00.754201 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.754959 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.756567 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.757063 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.757181 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:00.760821 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:00.760827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:00.760830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:00.760866 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:00.763325 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:00.764085 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:00.764090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:00.764092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:00.764221 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:00.764640 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:00.764644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:00.764646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:00.764963 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:00.765101 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:00.765830 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 11:38:00.766065 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260929 11:38:00.767468 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:00.749330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36405 (local address 127.6.9.194:49255)
0929 11:38:00.750016 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:00.750058 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:00.752003 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:00.752780 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:00.752792 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:00.753429 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:00.754017 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:00.754037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.755121 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.755125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:00.756414 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:00.756429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:00.757300 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:00.757308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:00.757535 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:00.758224 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:00.758266 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:00.760531 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:00.763465 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:00.763474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:00.763493 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:00.763965 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:00.764334 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:00.764339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:00.764343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:00.764532 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:00.765089 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:00.765097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:00.765471 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 11:38:00.765725 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 11:38:00.766275 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260929 11:38:00.768967 25825 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36405
I20260929 11:38:00.769328 25825 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:00.770270 25825 heartbeater.cc:511] Master 127.6.9.254:36405 requested a full tablet report, sending...
I20260929 11:38:00.771908 25502 ts_manager.cc:194] Registered new tserver with Master: 043bcda7f23c42b182f998eef7c4577b (127.6.9.194:38651)
WARNING: no policy specified for kudu/127.6.9.195@KRBTEST.COM; defaulting to no policy
I20260929 11:38:00.772810 25502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.194@KRBTEST.COM'} at 127.6.9.194:49255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
Entry for principal kudu/127.6.9.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab.
I20260929 11:38:00.802244 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:01.128790 25831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:01.129107 25831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:01.129213 25831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:01.137319 25831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:01.137471 25831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:01.137639 25831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:38:01.148597 25831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.195.keytab
--principal=kudu/127.6.9.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:36405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:01.150082 25831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:01.151549 25831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:01.166198 25836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:01.166225 25839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:01.166260 25837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:01.166679 25831 server_base.cc:1034] running on GCE node
I20260929 11:38:01.167558 25831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:01.168694 25831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:01.169920 25831 hybrid_clock.cc:648] HybridClock initialized: now 1790681881169870 us; error 43 us; skew 500 ppm
I20260929 11:38:01.170238 25831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:01 dist-test-slave-c0l9 krb5kdc[25457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681881, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:01.173624 25831 init.cc:377] Logged in from keytab as kudu/127.6.9.195@KRBTEST.COM (short username kudu)
I20260929 11:38:01.175182 25831 webserver.cc:460] Webserver started at http://127.6.9.195:39189/ using document root <none> and password file <none>
I20260929 11:38:01.175935 25831 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:01.176064 25831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:01.176389 25831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:01.179706 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "036a7d5549df4c4fa016c3180f203078"
format_stamp: "Formatted at 2026-09-29 11:38:01 on dist-test-slave-c0l9"
server_key: "e027bda834607a2e5b431bbcbd92e27e"
server_key_iv: "a9b59be2bcf18af94dc14798c7fa74f2"
server_key_version: "encryptionkey@0"
I20260929 11:38:01.180584 25831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "036a7d5549df4c4fa016c3180f203078"
format_stamp: "Formatted at 2026-09-29 11:38:01 on dist-test-slave-c0l9"
server_key: "e027bda834607a2e5b431bbcbd92e27e"
server_key_iv: "a9b59be2bcf18af94dc14798c7fa74f2"
server_key_version: "encryptionkey@0"
I20260929 11:38:01.186260 25831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:38:01.189908 25846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:01.191372 25831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:38:01.191548 25831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "036a7d5549df4c4fa016c3180f203078"
format_stamp: "Formatted at 2026-09-29 11:38:01 on dist-test-slave-c0l9"
server_key: "e027bda834607a2e5b431bbcbd92e27e"
server_key_iv: "a9b59be2bcf18af94dc14798c7fa74f2"
server_key_version: "encryptionkey@0"
I20260929 11:38:01.191746 25831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:01.230110 25831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:01.234741 25831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:01.235047 25831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:01.236094 25831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:01.237567 25831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:01.237649 25831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:01.237772 25831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:01.237874 25831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 11:38:01.241628 25691 heartbeater.cc:503] Master 127.6.9.254:36405 was elected leader, sending a full tablet report...
I20260929 11:38:01.269264 25831 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:46141
I20260929 11:38:01.269333 25958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:46141 every 8 connection(s)
I20260929 11:38:01.270663 25831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:38:01.276710 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 25831
I20260929 11:38:01.276937 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:38:01.277493 6183 external_mini_cluster.cc:1428] Setting key ca0d97821e4a50047169319697b8c854
Sep 29 11:38:01 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681881, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.195@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:01.292649 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.273366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:57847 (local address 127.6.9.254:36405)
0929 11:38:01.273879 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.273883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.276757 (+ 2874us) server_negotiation.cc:407] Connection header received
0929 11:38:01.277296 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.277303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.277407 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.277502 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:01.279318 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.280144 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.281987 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.282432 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.282587 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.286282 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:01.286287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:01.286291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:01.286329 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:01.289403 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.290236 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.290241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.290243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.290371 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.290830 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.290833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.290835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.291113 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:01.291264 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.292027 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.292268 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
I20260929 11:38:01.293661 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.273802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:36405 (local address 127.6.9.195:57847)
0929 11:38:01.274667 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.274716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.276986 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.277773 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.277782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.278492 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:01.279144 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.279163 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.280291 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.280295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.281813 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.281829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.282681 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.282690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.282959 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.283671 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:01.283696 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:01.286017 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:01.289582 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:01.289594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:01.289614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:01.290092 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:01.290484 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:01.290489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:01.290493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:01.290704 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:01.291246 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:01.291255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:01.291639 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.291882 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.292475 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
I20260929 11:38:01.295032 25959 heartbeater.cc:347] Connected to a master server at 127.6.9.254:36405
I20260929 11:38:01.295334 25959 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:01.296067 25959 heartbeater.cc:511] Master 127.6.9.254:36405 requested a full tablet report, sending...
I20260929 11:38:01.297741 25502 ts_manager.cc:194] Registered new tserver with Master: 036a7d5549df4c4fa016c3180f203078 (127.6.9.195:46141)
I20260929 11:38:01.298569 25502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.9.195@KRBTEST.COM'} at 127.6.9.195:57847
I20260929 11:38:01.305573 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:38:01.323601 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.309880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.6.9.254:36405)
0929 11:38:01.310109 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.310114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.310507 (+ 393us) server_negotiation.cc:407] Connection header received
0929 11:38:01.310684 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.310689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.310762 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.310863 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:01.312599 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.313402 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.314879 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.315318 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.315429 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.317997 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:01.318003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:01.318006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:01.318047 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:01.320717 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.321506 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.321511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.321514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.321668 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.322159 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.322163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.322165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.322565 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:01.322771 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.323169 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.323367 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 11:38:01.328279 25502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:38:01.335299 25502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:38:01.370371 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.348422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.6.9.193:44573)
0929 11:38:01.361375 (+ 12953us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.361385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.361644 (+ 259us) server_negotiation.cc:407] Connection header received
0929 11:38:01.362071 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.362079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.362366 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.362563 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.363529 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.365006 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.367630 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.369166 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.369299 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.369534 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.369863 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.370048 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12725,"thread_start_us":12974,"threads_started":1}
I20260929 11:38:01.370460 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.348302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:44573 (local address 127.0.0.1:53574)
0929 11:38:01.361466 (+ 13164us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.361516 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.361725 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.362888 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.362895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.362933 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.363324 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.363339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.365125 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.365130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.367454 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.367469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.369379 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.369386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.369582 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.369643 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.369801 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.369996 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12842,"mutex_wait_us":26,"thread_start_us":14544,"threads_started":1}
I20260929 11:38:01.371078 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.348779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:46141 (local address 127.0.0.1:59250)
0929 11:38:01.363187 (+ 14408us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.363208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.363333 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.364294 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.364300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.364329 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.364829 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.364842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.366697 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.366703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.368497 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.368509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.370277 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.370284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.370471 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.370502 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.370587 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.370710 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14270,"mutex_wait_us":60,"thread_start_us":12902,"threads_started":1}
I20260929 11:38:01.371948 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.349518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.6.9.195:46141)
0929 11:38:01.362919 (+ 13401us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.362927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.363464 (+ 537us) server_negotiation.cc:407] Connection header received
0929 11:38:01.363568 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.363575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.363826 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.364014 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.365027 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.366558 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.368639 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.369814 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.369934 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.370146 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.371003 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.371185 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13109,"thread_start_us":12956,"threads_started":1}
I20260929 11:38:01.372509 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.363661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.6.9.194:38651)
0929 11:38:01.364229 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.364237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.364905 (+ 668us) server_negotiation.cc:407] Connection header received
0929 11:38:01.365175 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.365184 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.365406 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.365598 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.366416 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.367504 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.369878 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.371224 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.371336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.371500 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.372035 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.372196 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":314,"threads_started":1}
I20260929 11:38:01.373009 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.363374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:38651 (local address 127.0.0.1:46786)
0929 11:38:01.364781 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.364803 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.364926 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.365807 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.365812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.365841 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.366236 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.366248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.367696 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.367701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.369713 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.369725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.371650 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.371656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.371822 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.371854 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.372554 (+ 700us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.372714 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":1083,"threads_started":1}
I20260929 11:38:01.374723 25626 tablet_service.cc:1467] Processing CreateTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec (DEFAULT_TABLE table=test-table [id=fbe37c01de7240b383f725a3f996d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:01.375979 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ee6dd25b1a484d8de2ba7d285014ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:01.375751 25894 tablet_service.cc:1467] Processing CreateTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec (DEFAULT_TABLE table=test-table [id=fbe37c01de7240b383f725a3f996d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:01.376483 25760 tablet_service.cc:1467] Processing CreateTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec (DEFAULT_TABLE table=test-table [id=fbe37c01de7240b383f725a3f996d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:01.377300 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ee6dd25b1a484d8de2ba7d285014ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:01.377841 25760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ee6dd25b1a484d8de2ba7d285014ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:01.388921 25978 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Bootstrap starting.
I20260929 11:38:01.392158 25979 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Bootstrap starting.
I20260929 11:38:01.393033 25978 tablet_bootstrap.cc:654] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:01.393024 25980 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Bootstrap starting.
I20260929 11:38:01.394351 25978 log.cc:824] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:01.396363 25979 tablet_bootstrap.cc:654] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:01.396404 25980 tablet_bootstrap.cc:654] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:01.397425 25979 log.cc:824] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:01.397609 25980 log.cc:824] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:01.397837 25978 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: No bootstrap required, opened a new log
I20260929 11:38:01.398188 25978 ts_tablet_manager.cc:1397] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 11:38:01.400502 25980 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: No bootstrap required, opened a new log
I20260929 11:38:01.400561 25979 tablet_bootstrap.cc:492] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: No bootstrap required, opened a new log
I20260929 11:38:01.400785 25980 ts_tablet_manager.cc:1397] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260929 11:38:01.400847 25979 ts_tablet_manager.cc:1397] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:38:01.407900 25980 raft_consensus.cc:348] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.408324 25980 raft_consensus.cc:374] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:01.408445 25980 raft_consensus.cc:729] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036a7d5549df4c4fa016c3180f203078, State: Initialized, Role: FOLLOWER
I20260929 11:38:01.409122 25980 consensus_queue.cc:260] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.409302 25978 raft_consensus.cc:348] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.409842 25978 raft_consensus.cc:374] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:01.409974 25978 raft_consensus.cc:729] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a3ff5f39ed4e8d8be0fe6946ee66cf, State: Initialized, Role: FOLLOWER
I20260929 11:38:01.410235 25959 heartbeater.cc:503] Master 127.6.9.254:36405 was elected leader, sending a full tablet report...
I20260929 11:38:01.410652 25978 consensus_queue.cc:260] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.410933 25980 ts_tablet_manager.cc:1428] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260929 11:38:01.411298 25979 raft_consensus.cc:348] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.411835 25979 raft_consensus.cc:374] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:01.411973 25979 raft_consensus.cc:729] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043bcda7f23c42b182f998eef7c4577b, State: Initialized, Role: FOLLOWER
I20260929 11:38:01.412700 25978 ts_tablet_manager.cc:1428] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 11:38:01.412726 25979 consensus_queue.cc:260] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.414149 25825 heartbeater.cc:503] Master 127.6.9.254:36405 was elected leader, sending a full tablet report...
I20260929 11:38:01.414901 25979 ts_tablet_manager.cc:1428] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 11:38:01.461942 25692 tablet.cc:2369] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:38:01.498373 25826 tablet.cc:2369] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:38:01.522347 25960 tablet.cc:2369] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:38:01.721695 25985 raft_consensus.cc:482] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:38:01.722013 25985 raft_consensus.cc:504] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.723732 25985 leader_election.cc:288] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 036a7d5549df4c4fa016c3180f203078 (127.6.9.195:46141), 043bcda7f23c42b182f998eef7c4577b (127.6.9.194:38651)
I20260929 11:38:01.729022 25984 raft_consensus.cc:482] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:38:01.729391 25984 raft_consensus.cc:504] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.730888 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.724260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:58197 (local address 127.6.9.195:46141)
0929 11:38:01.724472 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.724479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.725118 (+ 639us) server_negotiation.cc:407] Connection header received
0929 11:38:01.725307 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.725313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.725393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.725460 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.726051 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.726817 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.728933 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.729886 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.730010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.730116 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.730502 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.730651 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:38:01.731038 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.723992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:46141 (local address 127.6.9.193:58197)
0929 11:38:01.724972 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.725000 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.725161 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.725494 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.725498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.725523 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.725907 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.725917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.726967 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.726972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.728782 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.728797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.730091 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.730099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.730293 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.730361 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.730463 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.730608 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":586,"threads_started":1}
I20260929 11:38:01.731182 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.724355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:52849 (local address 127.6.9.194:38651)
0929 11:38:01.724682 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.724687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.725230 (+ 543us) server_negotiation.cc:407] Connection header received
0929 11:38:01.725384 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.725388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.725451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.725537 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.726191 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.726945 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.728842 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.730035 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.730173 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.730266 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.730734 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.730876 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 11:38:01.731452 25984 leader_election.cc:288] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a3ff5f39ed4e8d8be0fe6946ee66cf (127.6.9.193:44573), 043bcda7f23c42b182f998eef7c4577b (127.6.9.194:38651)
I20260929 11:38:01.731739 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.724183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:38651 (local address 127.6.9.193:52849)
0929 11:38:01.725117 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.725136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.725266 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.725572 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.725577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.725638 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.725995 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.726009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.727114 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.727118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.728642 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.728690 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.730289 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.730295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.730504 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.730534 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.731225 (+ 691us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.731372 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":35,"spinlock_wait_cycles":6912,"thread_start_us":648,"threads_started":1}
I20260929 11:38:01.732373 25914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036a7d5549df4c4fa016c3180f203078" is_pre_election: true
I20260929 11:38:01.732890 25914 raft_consensus.cc:2454] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06a3ff5f39ed4e8d8be0fe6946ee66cf in term 0.
I20260929 11:38:01.732810 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043bcda7f23c42b182f998eef7c4577b" is_pre_election: true
I20260929 11:38:01.733332 25780 raft_consensus.cc:2454] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06a3ff5f39ed4e8d8be0fe6946ee66cf in term 0.
I20260929 11:38:01.733655 25582 leader_election.cc:302] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 036a7d5549df4c4fa016c3180f203078, 06a3ff5f39ed4e8d8be0fe6946ee66cf; no voters:
I20260929 11:38:01.734098 25985 raft_consensus.cc:2781] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:38:01.734274 25985 raft_consensus.cc:482] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:38:01.734387 25985 raft_consensus.cc:3037] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:01.737809 25985 raft_consensus.cc:504] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.739064 25914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "036a7d5549df4c4fa016c3180f203078"
I20260929 11:38:01.739327 25914 raft_consensus.cc:3037] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:01.739353 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043bcda7f23c42b182f998eef7c4577b"
I20260929 11:38:01.739559 25780 raft_consensus.cc:3037] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:01.739938 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.732488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:38651 (local address 127.6.9.195:46733)
0929 11:38:01.733464 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.733529 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.733654 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.734146 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.734151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.734177 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.734523 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.734536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.735814 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.735819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.737461 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.737473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.739016 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.739022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.739207 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.739244 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.739351 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.739497 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":23,"thread_start_us":721,"threads_started":1}
I20260929 11:38:01.740494 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.732870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:46733 (local address 127.6.9.194:38651)
0929 11:38:01.733615 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.733620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.733638 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:38:01.733901 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.733905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.733974 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.734065 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.734678 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.735672 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.737601 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.738618 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.738765 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.738875 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.740151 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.740293 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
I20260929 11:38:01.740897 25985 leader_election.cc:288] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [CANDIDATE]: Term 1 election: Requested vote from peers 036a7d5549df4c4fa016c3180f203078 (127.6.9.195:46141), 043bcda7f23c42b182f998eef7c4577b (127.6.9.194:38651)
I20260929 11:38:01.741384 25779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "036a7d5549df4c4fa016c3180f203078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "043bcda7f23c42b182f998eef7c4577b" is_pre_election: true
I20260929 11:38:01.741600 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.732071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:44573 (local address 127.6.9.195:54567)
0929 11:38:01.732255 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:01.732274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:01.732380 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:01.734835 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:01.734839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:01.734864 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:01.735054 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.735063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.736447 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.736452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.738336 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:01.738350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.740830 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.740836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:01.741057 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.741093 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:38:01.741185 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 11:38:01.741339 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":24}
I20260929 11:38:01.742871 25780 raft_consensus.cc:2454] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a3ff5f39ed4e8d8be0fe6946ee66cf in term 1.
I20260929 11:38:01.743274 25580 leader_election.cc:302] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 043bcda7f23c42b182f998eef7c4577b, 06a3ff5f39ed4e8d8be0fe6946ee66cf; no voters:
I20260929 11:38:01.743569 25985 raft_consensus.cc:2781] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:01.743573 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.734371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:54567 (local address 127.6.9.193:44573)
0929 11:38:01.734549 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.734555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.734572 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:38:01.734637 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.734642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.734721 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.734806 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:01.735194 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.736274 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.738509 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.739708 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.739845 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.739946 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.743159 (+ 3213us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.743311 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 11:38:01.743929 25914 raft_consensus.cc:2454] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a3ff5f39ed4e8d8be0fe6946ee66cf in term 1.
I20260929 11:38:01.744284 25985 raft_consensus.cc:686] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 1 LEADER]: Becoming Leader. State: Replica: 06a3ff5f39ed4e8d8be0fe6946ee66cf, State: Running, Role: LEADER
I20260929 11:38:01.744534 25646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ee6dd25b1a484d8de2ba7d285014ec" candidate_uuid: "036a7d5549df4c4fa016c3180f203078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" is_pre_election: true
I20260929 11:38:01.744875 25985 consensus_queue.cc:237] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } }
I20260929 11:38:01.745568 25848 leader_election.cc:302] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 036a7d5549df4c4fa016c3180f203078; no voters: 043bcda7f23c42b182f998eef7c4577b, 06a3ff5f39ed4e8d8be0fe6946ee66cf
I20260929 11:38:01.748806 25984 raft_consensus.cc:2726] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 11:38:01.749450 25502 catalog_manager.cc:5495] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 06a3ff5f39ed4e8d8be0fe6946ee66cf (127.6.9.193). New cstate: current_term: 1 leader_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a3ff5f39ed4e8d8be0fe6946ee66cf" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 44573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 } health_report { overall_health: UNKNOWN } } }
I20260929 11:38:01.818895 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.813400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.6.9.193:44573)
0929 11:38:01.813612 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.813617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.813637 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:38:01.813752 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.813756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.813857 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.813995 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:01.814384 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.815132 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.816555 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.817052 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.817174 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.817717 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:01.818249 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:01.818400 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.818505 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.818599 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:38:01.827380 25914 raft_consensus.cc:1261] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 1 FOLLOWER]: Refusing update from remote peer 06a3ff5f39ed4e8d8be0fe6946ee66cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:01.827435 25780 raft_consensus.cc:1261] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 1 FOLLOWER]: Refusing update from remote peer 06a3ff5f39ed4e8d8be0fe6946ee66cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:01.828337 25990 consensus_queue.cc:1035] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "036a7d5549df4c4fa016c3180f203078" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 46141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:01.828789 25985 consensus_queue.cc:1035] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "043bcda7f23c42b182f998eef7c4577b" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 38651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:01.856383 25500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:38:01.856716 25500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:38:01.860884 25500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a3578f4726f46ebbe5e15015f589c7d: Sending DeleteTablet for 3 replicas of tablet e5ee6dd25b1a484d8de2ba7d285014ec
I20260929 11:38:01.862303 25894 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:01 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:59250
I20260929 11:38:01.862825 25626 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:01 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:53574
I20260929 11:38:01.863410 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25563
I20260929 11:38:01.863655 25760 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ee6dd25b1a484d8de2ba7d285014ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:01 UTC) from {username='kudu', principal='kudu/127.6.9.254@KRBTEST.COM'} at 127.0.0.1:46786
I20260929 11:38:01.864562 26005 tablet_replica.cc:330] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: stopping tablet replica
I20260929 11:38:01.865093 26005 raft_consensus.cc:2229] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:38:01.865664 26005 raft_consensus.cc:2258] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:01.868319 26005 ts_tablet_manager.cc:1905] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:01.875636 26004 tablet_replica.cc:330] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: stopping tablet replica
I20260929 11:38:01.876446 26004 raft_consensus.cc:2229] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:38:01.877131 26005 ts_tablet_manager.cc:1918] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:38:01.877332 26005 log.cc:1196] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/e5ee6dd25b1a484d8de2ba7d285014ec
I20260929 11:38:01.877446 26004 raft_consensus.cc:2258] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:01.877923 26005 ts_tablet_manager.cc:1939] T e5ee6dd25b1a484d8de2ba7d285014ec P 043bcda7f23c42b182f998eef7c4577b: Deleting consensus metadata
I20260929 11:38:01.878690 26003 tablet_replica.cc:330] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: stopping tablet replica
I20260929 11:38:01.879927 25487 catalog_manager.cc:4841] TS 043bcda7f23c42b182f998eef7c4577b (127.6.9.194:38651): tablet e5ee6dd25b1a484d8de2ba7d285014ec (table test-table [id=fbe37c01de7240b383f725a3f996d66e]) successfully deleted
I20260929 11:38:01.880437 26004 ts_tablet_manager.cc:1905] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:01.880677 26003 raft_consensus.cc:2229] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:38:01.881248 26003 raft_consensus.cc:2258] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:01.883555 26003 ts_tablet_manager.cc:1905] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:01.888805 26004 ts_tablet_manager.cc:1918] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:38:01.889027 26004 log.cc:1196] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-0/wal/wals/e5ee6dd25b1a484d8de2ba7d285014ec
Sep 29 11:38:01 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.193@KRBTEST.COM
I20260929 11:38:01.889623 26004 ts_tablet_manager.cc:1939] T e5ee6dd25b1a484d8de2ba7d285014ec P 06a3ff5f39ed4e8d8be0fe6946ee66cf: Deleting consensus metadata
I20260929 11:38:01.891778 25487 catalog_manager.cc:4841] TS 06a3ff5f39ed4e8d8be0fe6946ee66cf (127.6.9.193:44573): tablet e5ee6dd25b1a484d8de2ba7d285014ec (table test-table [id=fbe37c01de7240b383f725a3f996d66e]) successfully deleted
I20260929 11:38:01.892155 26003 ts_tablet_manager.cc:1918] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:38:01.892338 26003 log.cc:1196] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/e5ee6dd25b1a484d8de2ba7d285014ec
I20260929 11:38:01.892902 26003 ts_tablet_manager.cc:1939] T e5ee6dd25b1a484d8de2ba7d285014ec P 036a7d5549df4c4fa016c3180f203078: Deleting consensus metadata
I20260929 11:38:01.894665 25489 catalog_manager.cc:4841] TS 036a7d5549df4c4fa016c3180f203078 (127.6.9.195:46141): tablet e5ee6dd25b1a484d8de2ba7d285014ec (table test-table [id=fbe37c01de7240b383f725a3f996d66e]) successfully deleted
I20260929 11:38:01.898695 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:01.864505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.6.9.193:44573)
0929 11:38:01.864692 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:01.864698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:01.874690 (+ 9992us) server_negotiation.cc:407] Connection header received
0929 11:38:01.876064 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:01.876071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:01.876170 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:01.876262 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:01.880824 (+ 4562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.881960 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.884108 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:01.884866 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:01.884974 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:01.890873 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:01.890893 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:01.890913 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:01.890968 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:01.895212 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.896259 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.896267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.896270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.896483 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:01.897040 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:01.897045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:01.897048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:01.897590 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:01.897856 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:01.898214 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 11:38:01.898420 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:38:01.899420 25686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:01.998020 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25563
I20260929 11:38:02.013358 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25697
Sep 29 11:38:02 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.194@KRBTEST.COM
I20260929 11:38:02.028883 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:02.014255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.6.9.194:38651)
0929 11:38:02.014431 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:02.014435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:02.014454 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:38:02.014622 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:02.014627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:02.014692 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:02.014787 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:02.016022 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.016833 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.018120 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.018524 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.018645 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:02.022911 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:02.022929 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:02.022946 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:02.022987 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:02.025848 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:02.026670 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:02.026679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:02.026681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:02.026842 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:02.027270 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:02.027274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:02.027276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:02.027677 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:02.027924 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:02.028313 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 11:38:02.028512 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:38:02.029609 25820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:02.119501 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25697
I20260929 11:38:02.133911 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25831
Sep 29 11:38:02 dist-test-slave-c0l9 krb5kdc[25457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.195@KRBTEST.COM
I20260929 11:38:02.150223 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:02.134720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.6.9.195:46141)
0929 11:38:02.134917 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:02.134921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:02.134980 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:38:02.135168 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:02.135173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:02.135267 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:02.135347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:02.136727 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.137556 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.138877 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.139296 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.139398 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:02.144045 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:02.144063 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:02.144082 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:02.144127 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:02.147135 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:02.148059 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:02.148068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:02.148070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:02.148228 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:02.148745 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:02.148758 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:02.148761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:02.149185 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:02.149424 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:02.149759 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 11:38:02.149942 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:38:02.150964 25954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:02.242102 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25831
I20260929 11:38:02.256002 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 25470
I20260929 11:38:02.257210 25532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:02.370421 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 25470
2026-09-29T11:38:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4472 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T11:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:02Z Disabled control of system clock
I20260929 11:38:02.441797 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33845
--webserver_interface=127.6.9.254
--webserver_port=0
--builtin_ntp_servers=127.6.9.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:38:02.781509 26014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:02.781854 26014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:02.781958 26014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:02.789569 26014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:02.789709 26014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:02.789749 26014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:02.789829 26014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:02.789887 26014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:02.799618 26014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:02.801110 26014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:02.803354 26014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:02.815614 26019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:02.815645 26020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:02.815869 26022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:02.816905 26014 server_base.cc:1034] running on GCE node
I20260929 11:38:02.817752 26014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:02.819339 26014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:02.820569 26014 hybrid_clock.cc:648] HybridClock initialized: now 1790681882820521 us; error 43 us; skew 500 ppm
I20260929 11:38:02.820992 26014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:38:02.827863 26014 webserver.cc:460] Webserver started at http://127.6.9.254:37177/ using document root <none> and password file <none>
I20260929 11:38:02.828693 26014 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:02.828810 26014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:02.829157 26014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:02.832201 26014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "4dc783dd40b64be192c7e7e98857aadc"
format_stamp: "Formatted at 2026-09-29 11:38:02 on dist-test-slave-c0l9"
server_key: "092aa50253c6f08bbf169b826dc1f2a1"
server_key_iv: "5f5b596195a71b760a8cd7a7ec5912d1"
server_key_version: "encryptionkey@0"
I20260929 11:38:02.833093 26014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "4dc783dd40b64be192c7e7e98857aadc"
format_stamp: "Formatted at 2026-09-29 11:38:02 on dist-test-slave-c0l9"
server_key: "092aa50253c6f08bbf169b826dc1f2a1"
server_key_iv: "5f5b596195a71b760a8cd7a7ec5912d1"
server_key_version: "encryptionkey@0"
I20260929 11:38:02.838609 26014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:38:02.842372 26028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:02.844003 26014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:38:02.844226 26014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "4dc783dd40b64be192c7e7e98857aadc"
format_stamp: "Formatted at 2026-09-29 11:38:02 on dist-test-slave-c0l9"
server_key: "092aa50253c6f08bbf169b826dc1f2a1"
server_key_iv: "5f5b596195a71b760a8cd7a7ec5912d1"
server_key_version: "encryptionkey@0"
I20260929 11:38:02.844463 26014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:02.870113 26014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:02.871085 26014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:02.871392 26014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:02.889644 26014 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33845
I20260929 11:38:02.889665 26079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33845 every 8 connection(s)
I20260929 11:38:02.891080 26014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:02.895013 26080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:02.896445 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26014
I20260929 11:38:02.896634 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:02.897236 6183 external_mini_cluster.cc:1428] Setting key 23008f2879ecdaa1953cb1a847ebd88b
I20260929 11:38:02.908716 26080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: Bootstrap starting.
I20260929 11:38:02.911026 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:02.899335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.6.9.254:33845)
0929 11:38:02.900414 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:02.900426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:02.900471 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:38:02.902206 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:02.902230 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:02.902761 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:02.903151 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:02.903721 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.905042 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.906484 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:02.907199 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:02.907394 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:02.908604 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:02.908612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:02.908627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:02.909248 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:02.909331 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:02.909354 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:02.909458 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:02.909716 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:38:02.910031 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":368,"threads_started":1}
I20260929 11:38:02.912266 26080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:02.913383 26080 log.cc:824] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:02.916347 26080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: No bootstrap required, opened a new log
I20260929 11:38:02.924341 26080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } }
I20260929 11:38:02.924770 26080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:02.924878 26080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc783dd40b64be192c7e7e98857aadc, State: Initialized, Role: FOLLOWER
I20260929 11:38:02.925477 26080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } }
I20260929 11:38:02.925702 26080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:02.925837 26080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:02.925977 26080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:02.928109 26080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } }
I20260929 11:38:02.928597 26080 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc783dd40b64be192c7e7e98857aadc; no voters:
I20260929 11:38:02.929286 26080 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:02.929462 26085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:02.930115 26085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [term 1 LEADER]: Becoming Leader. State: Replica: 4dc783dd40b64be192c7e7e98857aadc, State: Running, Role: LEADER
I20260929 11:38:02.930748 26085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } }
I20260929 11:38:02.931408 26080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:02.933820 26086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc783dd40b64be192c7e7e98857aadc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } } }
I20260929 11:38:02.933874 26087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc783dd40b64be192c7e7e98857aadc. Latest consensus state: current_term: 1 leader_uuid: "4dc783dd40b64be192c7e7e98857aadc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc783dd40b64be192c7e7e98857aadc" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33845 } } }
I20260929 11:38:02.934264 26086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:02.934269 26087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:02.936174 26091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:02.942660 26091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:02.952335 26091 catalog_manager.cc:1347] Generated new cluster ID: 0c6328cb068e4a8aa7656cf9318c3233
I20260929 11:38:02.952461 26091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:02.963822 26091 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:02.965257 26091 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:02.982795 26091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: Generated new TSK 0
I20260929 11:38:02.983732 26091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:02.993793 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.193:0
--local_ip_for_outbound_sockets=127.6.9.193
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--builtin_ntp_servers=127.6.9.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:38:03.369491 26104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:03.369800 26104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:03.369892 26104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:03.377678 26104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:03.377810 26104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:03.377924 26104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.193
I20260929 11:38:03.387773 26104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.9.193
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.193
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:03.389075 26104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:03.390854 26104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:03.404253 26109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:03.404191 26112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:03.404227 26110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:03.404604 26104 server_base.cc:1034] running on GCE node
I20260929 11:38:03.405671 26104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:03.406636 26104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:03.407866 26104 hybrid_clock.cc:648] HybridClock initialized: now 1790681883407809 us; error 44 us; skew 500 ppm
I20260929 11:38:03.408231 26104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:38:03.410682 26104 webserver.cc:460] Webserver started at http://127.6.9.193:39285/ using document root <none> and password file <none>
I20260929 11:38:03.411459 26104 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:03.411551 26104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:03.411880 26104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:03.414870 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data/instance:
uuid: "b486461209214a59a94b6144234b3b40"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "9d4bd19bceaaa2ee0e7f6cc19968f836"
server_key_iv: "2030dbb282f220569a6c13ea72284ef8"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.415689 26104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance:
uuid: "b486461209214a59a94b6144234b3b40"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "9d4bd19bceaaa2ee0e7f6cc19968f836"
server_key_iv: "2030dbb282f220569a6c13ea72284ef8"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.421016 26104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:38:03.424733 26118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.426131 26104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:03.426294 26104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal
uuid: "b486461209214a59a94b6144234b3b40"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "9d4bd19bceaaa2ee0e7f6cc19968f836"
server_key_iv: "2030dbb282f220569a6c13ea72284ef8"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.426496 26104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:03.445171 26104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:03.446090 26104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:03.446379 26104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:03.447430 26104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:03.449210 26104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:03.449306 26104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.449414 26104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:03.449488 26104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.480156 26104 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.193:45549
I20260929 11:38:03.480254 26230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.193:45549 every 8 connection(s)
I20260929 11:38:03.481725 26104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/data/info.pb
I20260929 11:38:03.486485 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26104
I20260929 11:38:03.486671 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-0/wal/instance
I20260929 11:38:03.487097 6183 external_mini_cluster.cc:1428] Setting key b761fbb1e48088c4245546ebb342d21c
I20260929 11:38:03.489996 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.194:0
--local_ip_for_outbound_sockets=127.6.9.194
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--builtin_ntp_servers=127.6.9.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:38:03.495611 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:03.484422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.193:35753 (local address 127.6.9.254:33845)
0929 11:38:03.484926 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:03.484933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:03.486206 (+ 1273us) server_negotiation.cc:407] Connection header received
0929 11:38:03.488246 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:03.488251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:03.488335 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:03.488435 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:03.489772 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.490700 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.492616 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.493086 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.493219 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:03.494715 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:03.494720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:03.494723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:03.494745 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:03.494787 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:03.494799 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:03.494877 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:03.495119 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 11:38:03.495229 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260929 11:38:03.496722 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:03.485010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33845 (local address 127.6.9.193:35753)
0929 11:38:03.485996 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:03.486040 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:03.487923 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:03.488756 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:03.488769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:03.488904 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:03.489588 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:03.489607 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.490888 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.490892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:03.492415 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:03.492433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.493355 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.493363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:03.493608 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:03.494354 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:38:03.494356 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:03.494403 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:38:03.494494 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:03.494891 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:03.494905 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:03.494913 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:38:03.495081 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 11:38:03.495541 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
I20260929 11:38:03.498396 26231 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33845
I20260929 11:38:03.498716 26231 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:03.499513 26231 heartbeater.cc:511] Master 127.6.9.254:33845 requested a full tablet report, sending...
I20260929 11:38:03.501827 26045 ts_manager.cc:194] Registered new tserver with Master: b486461209214a59a94b6144234b3b40 (127.6.9.193:45549)
I20260929 11:38:03.503490 26045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.193:35753
W20260929 11:38:03.825704 26235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:03.826035 26235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:03.826166 26235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:03.833772 26235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:03.833907 26235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:03.834060 26235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.194
I20260929 11:38:03.843613 26235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.9.194
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.194
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:03.844982 26235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:03.846837 26235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:03.860251 26243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:03.860311 26240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:03.860288 26241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:03.861171 26235 server_base.cc:1034] running on GCE node
I20260929 11:38:03.861810 26235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:03.862811 26235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:03.864020 26235 hybrid_clock.cc:648] HybridClock initialized: now 1790681883863973 us; error 41 us; skew 500 ppm
I20260929 11:38:03.864375 26235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:38:03.866778 26235 webserver.cc:460] Webserver started at http://127.6.9.194:44479/ using document root <none> and password file <none>
I20260929 11:38:03.867513 26235 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:03.867615 26235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:03.867935 26235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:03.871021 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data/instance:
uuid: "203799a96435471c87c3dc42cb644113"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "3df1ac0570e09726d0d772422dc5e675"
server_key_iv: "be6f79e8db26f82aa855dcba5dd9c668"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.871853 26235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance:
uuid: "203799a96435471c87c3dc42cb644113"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "3df1ac0570e09726d0d772422dc5e675"
server_key_iv: "be6f79e8db26f82aa855dcba5dd9c668"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.877210 26235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 11:38:03.880856 26249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.882208 26235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:03.882364 26235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal
uuid: "203799a96435471c87c3dc42cb644113"
format_stamp: "Formatted at 2026-09-29 11:38:03 on dist-test-slave-c0l9"
server_key: "3df1ac0570e09726d0d772422dc5e675"
server_key_iv: "be6f79e8db26f82aa855dcba5dd9c668"
server_key_version: "encryptionkey@0"
I20260929 11:38:03.882571 26235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:03.907573 26235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:03.908604 26235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:03.908972 26235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:03.910009 26235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:03.911571 26235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:03.911669 26235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.911784 26235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:03.911857 26235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:03.943152 26235 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.194:44633
I20260929 11:38:03.943216 26361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.194:44633 every 8 connection(s)
I20260929 11:38:03.944476 26235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/data/info.pb
I20260929 11:38:03.945868 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26235
I20260929 11:38:03.946017 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal/instance
I20260929 11:38:03.946448 6183 external_mini_cluster.cc:1428] Setting key 17db862f5acabd0cfafd586807efcc5f
I20260929 11:38:03.949440 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.9.195:0
--local_ip_for_outbound_sockets=127.6.9.195
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--builtin_ntp_servers=127.6.9.212:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:38:03.959426 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:03.948481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.194:33813 (local address 127.6.9.254:33845)
0929 11:38:03.948723 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:03.948731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:03.950498 (+ 1767us) server_negotiation.cc:407] Connection header received
0929 11:38:03.952325 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:03.952331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:03.952421 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:03.952531 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:03.953741 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.954721 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.956459 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.956933 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.957058 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:03.958584 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:03.958589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:03.958593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:03.958614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:03.958649 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:03.958662 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:03.958742 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:03.958980 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 11:38:03.959108 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:38:03.960577 26364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:03.949246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33845 (local address 127.6.9.194:33813)
0929 11:38:03.950313 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:03.950359 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:03.952027 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:03.952814 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:03.952824 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:03.952932 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:03.953549 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:03.953568 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.954920 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.954925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:03.956271 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:03.956285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:03.957190 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:03.957199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:03.957439 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:03.958187 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:38:03.958190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:03.958234 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:38:03.958350 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:03.958757 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:03.958770 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:03.958779 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:38:03.958949 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 11:38:03.959410 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":320,"threads_started":1}
I20260929 11:38:03.962322 26362 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33845
I20260929 11:38:03.962641 26362 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:03.963439 26362 heartbeater.cc:511] Master 127.6.9.254:33845 requested a full tablet report, sending...
I20260929 11:38:03.965157 26045 ts_manager.cc:194] Registered new tserver with Master: 203799a96435471c87c3dc42cb644113 (127.6.9.194:44633)
I20260929 11:38:03.966274 26045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.194:33813
W20260929 11:38:04.282843 26365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:04.283161 26365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:04.283252 26365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:04.292100 26365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:04.292227 26365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:04.292346 26365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.9.195
I20260929 11:38:04.302076 26365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.9.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.9.195
--webserver_port=0
--tserver_master_addrs=127.6.9.254:33845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.9.195
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:04.303376 26365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:04.305229 26365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:04.318425 26371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:04.318456 26370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:04.318382 26373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:04.319303 26365 server_base.cc:1034] running on GCE node
I20260929 11:38:04.319911 26365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:04.320899 26365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:04.322137 26365 hybrid_clock.cc:648] HybridClock initialized: now 1790681884322074 us; error 49 us; skew 500 ppm
I20260929 11:38:04.322481 26365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:38:04.324940 26365 webserver.cc:460] Webserver started at http://127.6.9.195:34545/ using document root <none> and password file <none>
I20260929 11:38:04.325644 26365 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:04.325754 26365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:04.326097 26365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:04.329123 26365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data/instance:
uuid: "48e834ee7aad4157b23ced39bcdab8bd"
format_stamp: "Formatted at 2026-09-29 11:38:04 on dist-test-slave-c0l9"
server_key: "af3cfbcab7d6e813e10c79be113a2f48"
server_key_iv: "e0058af6a4c46b1c71ecc82c30ff1715"
server_key_version: "encryptionkey@0"
I20260929 11:38:04.329962 26365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance:
uuid: "48e834ee7aad4157b23ced39bcdab8bd"
format_stamp: "Formatted at 2026-09-29 11:38:04 on dist-test-slave-c0l9"
server_key: "af3cfbcab7d6e813e10c79be113a2f48"
server_key_iv: "e0058af6a4c46b1c71ecc82c30ff1715"
server_key_version: "encryptionkey@0"
I20260929 11:38:04.335220 26365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 11:38:04.338820 26379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:04.340147 26365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:38:04.340315 26365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal
uuid: "48e834ee7aad4157b23ced39bcdab8bd"
format_stamp: "Formatted at 2026-09-29 11:38:04 on dist-test-slave-c0l9"
server_key: "af3cfbcab7d6e813e10c79be113a2f48"
server_key_iv: "e0058af6a4c46b1c71ecc82c30ff1715"
server_key_version: "encryptionkey@0"
I20260929 11:38:04.340538 26365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:04.379171 26365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:04.380086 26365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:04.380362 26365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:04.381438 26365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:38:04.383000 26365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:38:04.383098 26365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:04.383213 26365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:38:04.383284 26365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:04.414279 26365 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.195:36433
I20260929 11:38:04.414338 26491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.195:36433 every 8 connection(s)
I20260929 11:38:04.415550 26365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/data/info.pb
I20260929 11:38:04.416478 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26365
I20260929 11:38:04.416643 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal/instance
I20260929 11:38:04.417165 6183 external_mini_cluster.cc:1428] Setting key 8516d1e09dfcc239cb2653943b100562
I20260929 11:38:04.427675 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.418234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:51101 (local address 127.6.9.254:33845)
0929 11:38:04.418420 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.418426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.420234 (+ 1808us) server_negotiation.cc:407] Connection header received
0929 11:38:04.422035 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.422039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.422106 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.422187 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:04.423147 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.423871 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.425357 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.425746 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.425854 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.427040 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:04.427044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:04.427048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:04.427065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:04.427096 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:04.427108 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:04.427175 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.427331 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.427409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:38:04.428618 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.418904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33845 (local address 127.6.9.195:51101)
0929 11:38:04.420070 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.420117 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.421799 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.422370 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.422378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.422465 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:04.423029 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.423044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.423985 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.423989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.425246 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.425258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.425912 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.425916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.426107 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.426740 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:38:04.426743 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:04.426782 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:38:04.426866 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:04.427179 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:04.427185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:04.427190 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.427319 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.427603 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":344,"threads_started":1}
I20260929 11:38:04.429926 26492 heartbeater.cc:347] Connected to a master server at 127.6.9.254:33845
I20260929 11:38:04.430265 26492 heartbeater.cc:464] Registering TS with master...
I20260929 11:38:04.431219 26492 heartbeater.cc:511] Master 127.6.9.254:33845 requested a full tablet report, sending...
I20260929 11:38:04.432731 26045 ts_manager.cc:194] Registered new tserver with Master: 48e834ee7aad4157b23ced39bcdab8bd (127.6.9.195:36433)
I20260929 11:38:04.433521 26045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.9.195:51101
I20260929 11:38:04.445209 6183 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:38:04.455543 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.449198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.6.9.254:33845)
0929 11:38:04.449348 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.449353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.449798 (+ 445us) server_negotiation.cc:407] Connection header received
0929 11:38:04.449967 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.449971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.450049 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.450134 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:04.450620 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.451453 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.453038 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.453496 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.453638 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.454833 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:04.454838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:04.454841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:04.454860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:04.454898 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:04.454911 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:04.454987 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.455150 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.455249 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:38:04.459525 26045 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:38:04.466235 26045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:38:04.507014 26231 heartbeater.cc:503] Master 127.6.9.254:33845 was elected leader, sending a full tablet report...
I20260929 11:38:04.514362 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.6.9.195:36433)
0929 11:38:04.505336 (+ 26796us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.505345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.505383 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:38:04.505487 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.505495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.505784 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.506007 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:04.506933 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.508531 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.511046 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513130 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513275 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.513571 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.513812 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.513988 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26610,"thread_start_us":25857,"threads_started":1}
I20260929 11:38:04.514369 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.195:36433 (local address 127.0.0.1:44728)
0929 11:38:04.502478 (+ 24401us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.502540 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.502758 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.506049 (+ 3291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.506057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.506091 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:04.506544 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.506557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.508726 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.508731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.510759 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.510774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513337 (+ 2563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.513560 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.513616 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.513752 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.513968 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24059,"mutex_wait_us":165,"thread_start_us":26970,"threads_started":1}
I20260929 11:38:04.514917 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:44633 (local address 127.0.0.1:59342)
0929 11:38:04.505238 (+ 26741us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.505266 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.505421 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.506305 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.506313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.506337 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:04.506922 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.506935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.509154 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.509159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.511412 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.511428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513697 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.513989 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.514061 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.514142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.514295 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26558,"mutex_wait_us":106,"thread_start_us":25065,"threads_started":1}
I20260929 11:38:04.514951 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.6.9.194:44633)
0929 11:38:04.501552 (+ 22559us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.501565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.505397 (+ 3832us) server_negotiation.cc:407] Connection header received
0929 11:38:04.505637 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.505646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.505825 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.505967 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:04.507117 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.509020 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.511679 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513168 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513301 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.513537 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.514458 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.514627 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22332,"thread_start_us":22583,"threads_started":1}
I20260929 11:38:04.515511 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.6.9.193:45549)
0929 11:38:04.503522 (+ 25321us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.503531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.505524 (+ 1993us) server_negotiation.cc:407] Connection header received
0929 11:38:04.505663 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.505670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.505837 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.506007 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:04.507034 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.508632 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.511679 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513168 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513318 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.513583 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.514261 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.514778 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25166,"thread_start_us":25051,"threads_started":1}
I20260929 11:38:04.516156 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.478077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47812)
0929 11:38:04.505391 (+ 27314us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.505426 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.505533 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.506272 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.506276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.506297 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:04.506823 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.506837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.508811 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.508819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.511421 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.511430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.513708 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.513714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.513994 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.514061 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.514142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.515835 (+ 1693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27210,"mutex_wait_us":2562,"thread_start_us":26031,"threads_started":1}
I20260929 11:38:04.518203 26427 tablet_service.cc:1467] Processing CreateTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 (DEFAULT_TABLE table=test-table [id=d1c51d3b93634f0d960eee818147e20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:04.518204 26297 tablet_service.cc:1467] Processing CreateTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 (DEFAULT_TABLE table=test-table [id=d1c51d3b93634f0d960eee818147e20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:04.519381 26427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ecfee31ecd4e428d306efd444a2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:04.519383 26297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ecfee31ecd4e428d306efd444a2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:04.524224 26166 tablet_service.cc:1467] Processing CreateTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 (DEFAULT_TABLE table=test-table [id=d1c51d3b93634f0d960eee818147e20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:38:04.525868 26166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ecfee31ecd4e428d306efd444a2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:04.530872 26510 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Bootstrap starting.
I20260929 11:38:04.534142 26511 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Bootstrap starting.
I20260929 11:38:04.536248 26510 tablet_bootstrap.cc:654] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:04.537570 26511 tablet_bootstrap.cc:654] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:04.537533 26510 log.cc:824] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:04.538652 26511 log.cc:824] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:04.540723 26510 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: No bootstrap required, opened a new log
I20260929 11:38:04.540738 26514 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Bootstrap starting.
I20260929 11:38:04.541136 26510 ts_tablet_manager.cc:1397] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260929 11:38:04.541798 26511 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: No bootstrap required, opened a new log
I20260929 11:38:04.542065 26511 ts_tablet_manager.cc:1397] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 11:38:04.543650 26514 tablet_bootstrap.cc:654] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:04.544693 26514 log.cc:824] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:04.548408 26514 tablet_bootstrap.cc:492] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: No bootstrap required, opened a new log
I20260929 11:38:04.548652 26514 ts_tablet_manager.cc:1397] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 11:38:04.551860 26510 raft_consensus.cc:348] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.552357 26510 raft_consensus.cc:374] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:04.552465 26510 raft_consensus.cc:729] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e834ee7aad4157b23ced39bcdab8bd, State: Initialized, Role: FOLLOWER
I20260929 11:38:04.552560 26511 raft_consensus.cc:348] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.553022 26511 raft_consensus.cc:374] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:04.553126 26511 raft_consensus.cc:729] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203799a96435471c87c3dc42cb644113, State: Initialized, Role: FOLLOWER
I20260929 11:38:04.553176 26510 consensus_queue.cc:260] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.553697 26511 consensus_queue.cc:260] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.554461 26492 heartbeater.cc:503] Master 127.6.9.254:33845 was elected leader, sending a full tablet report...
I20260929 11:38:04.556772 26510 ts_tablet_manager.cc:1428] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 11:38:04.556548 26514 raft_consensus.cc:348] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.557024 26514 raft_consensus.cc:374] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:04.557120 26514 raft_consensus.cc:729] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b486461209214a59a94b6144234b3b40, State: Initialized, Role: FOLLOWER
I20260929 11:38:04.557744 26362 heartbeater.cc:503] Master 127.6.9.254:33845 was elected leader, sending a full tablet report...
I20260929 11:38:04.558270 26511 ts_tablet_manager.cc:1428] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Time spent starting tablet: real 0.016s user 0.006s sys 0.011s
I20260929 11:38:04.558693 26514 consensus_queue.cc:260] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.560456 26514 ts_tablet_manager.cc:1428] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20260929 11:38:04.667332 26493 tablet.cc:2369] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:38:04.696640 26363 tablet.cc:2369] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:38:04.734207 26232 tablet.cc:2369] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:38:04.798039 26516 raft_consensus.cc:482] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:38:04.798350 26516 raft_consensus.cc:504] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.800004 26516 leader_election.cc:288] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b486461209214a59a94b6144234b3b40 (127.6.9.193:45549), 203799a96435471c87c3dc42cb644113 (127.6.9.194:44633)
I20260929 11:38:04.806461 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.800307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.194:44633 (local address 127.6.9.195:47455)
0929 11:38:04.800481 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.800500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.800630 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.801192 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.801196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.801224 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:04.801405 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.801413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.802670 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.802675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.804313 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.804326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.805757 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.805763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.805920 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.805945 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.806022 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.806162 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 11:38:04.806456 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.800714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:47455 (local address 127.6.9.194:44633)
0929 11:38:04.800897 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.800901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.800916 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:38:04.800976 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.800980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.801047 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.801124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:04.801579 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.802532 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.804482 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.805616 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.805728 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.805833 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.806075 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.806200 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:38:04.807111 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.800530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.6.9.195:43703)
0929 11:38:04.801296 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:04.801310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:04.801447 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:04.801889 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:04.801893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:04.801920 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:38:04.802272 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.802283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.803593 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.803597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.805285 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:04.805299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.806428 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.806434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:04.806621 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.806665 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 11:38:04.806749 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 11:38:04.806869 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":416,"threads_started":1}
I20260929 11:38:04.807163 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.800585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.9.195:43703 (local address 127.6.9.193:45549)
0929 11:38:04.800816 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.800822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.801403 (+ 581us) server_negotiation.cc:407] Connection header received
0929 11:38:04.801669 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.801675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.801769 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.801859 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:38:04.802422 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.803406 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.805490 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.806261 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.806378 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.806473 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.806792 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.806930 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 11:38:04.807621 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ecfee31ecd4e428d306efd444a2a55" candidate_uuid: "48e834ee7aad4157b23ced39bcdab8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "203799a96435471c87c3dc42cb644113" is_pre_election: true
I20260929 11:38:04.808163 26186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ecfee31ecd4e428d306efd444a2a55" candidate_uuid: "48e834ee7aad4157b23ced39bcdab8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486461209214a59a94b6144234b3b40" is_pre_election: true
I20260929 11:38:04.808290 26317 raft_consensus.cc:2454] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e834ee7aad4157b23ced39bcdab8bd in term 0.
I20260929 11:38:04.808720 26186 raft_consensus.cc:2454] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e834ee7aad4157b23ced39bcdab8bd in term 0.
I20260929 11:38:04.809015 26383 leader_election.cc:302] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 203799a96435471c87c3dc42cb644113, 48e834ee7aad4157b23ced39bcdab8bd; no voters:
I20260929 11:38:04.809456 26516 raft_consensus.cc:2781] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:38:04.809579 26516 raft_consensus.cc:482] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:38:04.809672 26516 raft_consensus.cc:3037] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:04.811823 26516 raft_consensus.cc:504] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.812454 26516 leader_election.cc:288] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [CANDIDATE]: Term 1 election: Requested vote from peers b486461209214a59a94b6144234b3b40 (127.6.9.193:45549), 203799a96435471c87c3dc42cb644113 (127.6.9.194:44633)
I20260929 11:38:04.812964 26186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ecfee31ecd4e428d306efd444a2a55" candidate_uuid: "48e834ee7aad4157b23ced39bcdab8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b486461209214a59a94b6144234b3b40"
I20260929 11:38:04.813146 26186 raft_consensus.cc:3037] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:04.813114 26317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ecfee31ecd4e428d306efd444a2a55" candidate_uuid: "48e834ee7aad4157b23ced39bcdab8bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "203799a96435471c87c3dc42cb644113"
I20260929 11:38:04.813328 26317 raft_consensus.cc:3037] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:04.815363 26186 raft_consensus.cc:2454] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e834ee7aad4157b23ced39bcdab8bd in term 1.
I20260929 11:38:04.815665 26317 raft_consensus.cc:2454] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e834ee7aad4157b23ced39bcdab8bd in term 1.
I20260929 11:38:04.815768 26380 leader_election.cc:302] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e834ee7aad4157b23ced39bcdab8bd, b486461209214a59a94b6144234b3b40; no voters:
I20260929 11:38:04.816098 26516 raft_consensus.cc:2781] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:04.816483 26516 raft_consensus.cc:686] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 1 LEADER]: Becoming Leader. State: Replica: 48e834ee7aad4157b23ced39bcdab8bd, State: Running, Role: LEADER
I20260929 11:38:04.817176 26516 consensus_queue.cc:237] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } }
I20260929 11:38:04.821940 26042 catalog_manager.cc:5495] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e834ee7aad4157b23ced39bcdab8bd (127.6.9.195). New cstate: current_term: 1 leader_uuid: "48e834ee7aad4157b23ced39bcdab8bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e834ee7aad4157b23ced39bcdab8bd" member_type: VOTER last_known_addr { host: "127.6.9.195" port: 36433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 } health_report { overall_health: UNKNOWN } } }
I20260929 11:38:04.860608 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.855247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.6.9.195:36433)
0929 11:38:04.855450 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.855456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.855476 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:38:04.855711 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.855715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.855806 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.855967 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:04.856342 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.857177 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.858538 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.858952 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.859054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.859497 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:04.860005 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:04.860127 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.860162 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.860228 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:38:04.868589 26186 raft_consensus.cc:1261] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 1 FOLLOWER]: Refusing update from remote peer 48e834ee7aad4157b23ced39bcdab8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:04.868592 26317 raft_consensus.cc:1261] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 1 FOLLOWER]: Refusing update from remote peer 48e834ee7aad4157b23ced39bcdab8bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:04.869633 26520 consensus_queue.cc:1035] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "203799a96435471c87c3dc42cb644113" member_type: VOTER last_known_addr { host: "127.6.9.194" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:04.869984 26516 consensus_queue.cc:1035] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b486461209214a59a94b6144234b3b40" member_type: VOTER last_known_addr { host: "127.6.9.193" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:04.884519 26524 mvcc.cc:205] Tried to move back new op lower bound from 7334633000412090368 to 7334633000214691840. Current Snapshot: MvccSnapshot[applied={T|T < 7334633000412090368}]
I20260929 11:38:04.900189 26044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:38:04.900496 26044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:38:04.904246 26044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dc783dd40b64be192c7e7e98857aadc: Sending DeleteTablet for 3 replicas of tablet 54ecfee31ecd4e428d306efd444a2a55
I20260929 11:38:04.905417 26427 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:04 UTC) from {username='slave'} at 127.0.0.1:44728
I20260929 11:38:04.905455 26166 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:04 UTC) from {username='slave'} at 127.0.0.1:47812
I20260929 11:38:04.905686 26297 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ecfee31ecd4e428d306efd444a2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:38:04 UTC) from {username='slave'} at 127.0.0.1:59342
I20260929 11:38:04.909061 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26104
I20260929 11:38:04.920186 26535 tablet_replica.cc:330] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: stopping tablet replica
I20260929 11:38:04.920763 26535 raft_consensus.cc:2229] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:38:04.921379 26535 raft_consensus.cc:2258] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:04.923759 26535 ts_tablet_manager.cc:1905] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:04.931388 26535 ts_tablet_manager.cc:1918] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:38:04.931561 26535 log.cc:1196] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-1/wal/wals/54ecfee31ecd4e428d306efd444a2a55
I20260929 11:38:04.932163 26535 ts_tablet_manager.cc:1939] T 54ecfee31ecd4e428d306efd444a2a55 P 203799a96435471c87c3dc42cb644113: Deleting consensus metadata
I20260929 11:38:04.935721 26032 catalog_manager.cc:4841] TS 203799a96435471c87c3dc42cb644113 (127.6.9.194:44633): tablet 54ecfee31ecd4e428d306efd444a2a55 (table test-table [id=d1c51d3b93634f0d960eee818147e20e]) successfully deleted
I20260929 11:38:04.935966 26533 tablet_replica.cc:330] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: stopping tablet replica
I20260929 11:38:04.936448 26533 raft_consensus.cc:2229] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:38:04.937134 26533 raft_consensus.cc:2258] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:04.938004 26534 tablet_replica.cc:330] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: stopping tablet replica
I20260929 11:38:04.938557 26534 raft_consensus.cc:2229] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:38:04.939217 26505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:04.910385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.6.9.193:45549)
0929 11:38:04.910571 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:04.910578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:04.930367 (+ 19789us) server_negotiation.cc:407] Connection header received
0929 11:38:04.930939 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:04.930946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:04.931076 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:04.931156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:04.931818 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.933035 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.934991 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:04.935811 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:04.935939 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:04.937403 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:04.937414 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:04.937430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:04.938275 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:04.938338 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:04.938366 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:04.938459 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:04.938760 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 11:38:04.938882 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:38:04.939610 26534 raft_consensus.cc:2258] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:38:04.939633 26533 ts_tablet_manager.cc:1905] T 54ecfee31ecd4e428d306efd444a2a55 P b486461209214a59a94b6144234b3b40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:04.940436 26226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:04.941805 26534 ts_tablet_manager.cc:1905] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:38:04.951143 26534 ts_tablet_manager.cc:1918] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:38:04.951373 26534 log.cc:1196] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Deleting WAL directory at /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681566518919-6183-0/minicluster-data/ts-2/wal/wals/54ecfee31ecd4e428d306efd444a2a55
I20260929 11:38:04.951992 26534 ts_tablet_manager.cc:1939] T 54ecfee31ecd4e428d306efd444a2a55 P 48e834ee7aad4157b23ced39bcdab8bd: Deleting consensus metadata
I20260929 11:38:04.954219 26031 catalog_manager.cc:4841] TS 48e834ee7aad4157b23ced39bcdab8bd (127.6.9.195:36433): tablet 54ecfee31ecd4e428d306efd444a2a55 (table test-table [id=d1c51d3b93634f0d960eee818147e20e]) successfully deleted
I20260929 11:38:05.033780 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26104
I20260929 11:38:05.048862 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26235
W20260929 11:38:05.048776 26029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.9.193:45549 (error 108)
I20260929 11:38:05.051380 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.050273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47832)
0929 11:38:05.051028 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.051178 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":535,"threads_started":1}
W20260929 11:38:05.051651 26029 catalog_manager.cc:4584] TS b486461209214a59a94b6144234b3b40 (127.6.9.193:45549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54ecfee31ecd4e428d306efd444a2a55: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
I20260929 11:38:05.057242 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.049932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.6.9.194:44633)
0929 11:38:05.050092 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:05.050096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:05.050112 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:38:05.050166 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:05.050169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:05.050250 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:05.050343 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:05.050778 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:05.051843 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:05.053814 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:05.054239 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:05.054336 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:05.055615 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:05.055625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:05.055641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:05.056300 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:05.056387 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:05.056419 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:05.056568 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:05.056723 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 11:38:05.056831 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:38:05.057965 26357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:05.105484 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.104913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47842)
0929 11:38:05.105172 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.105291 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 11:38:05.107038 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.106556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47852)
0929 11:38:05.106744 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.106845 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:38:05.148738 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26235
I20260929 11:38:05.167958 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.167432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47862)
0929 11:38:05.167650 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.167762 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:38:05.169164 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.168748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47870)
0929 11:38:05.168905 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.168982 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 11:38:05.169474 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26365
I20260929 11:38:05.176506 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.170230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.6.9.195:36433)
0929 11:38:05.170408 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:05.170415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:05.170433 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:38:05.170617 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:05.170622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:05.170708 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:05.170794 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:05.171241 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:05.172033 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:05.173380 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:05.173846 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:05.173954 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:05.175055 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:05.175065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:05.175080 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:05.175708 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:05.175760 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:05.175780 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:05.175877 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:05.176031 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 11:38:05.176155 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:38:05.177315 26487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:05.249966 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.249446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47880)
0929 11:38:05.249708 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.249805 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:38:05.251044 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.250648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47894)
0929 11:38:05.250830 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.250914 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 11:38:05.302225 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26365
I20260929 11:38:05.318310 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26014
I20260929 11:38:05.319504 26075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:05.432272 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26014
I20260929 11:38:05.432336 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:05.431413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.193:45549 (local address 127.0.0.1:47906)
0929 11:38:05.431898 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:05.432056 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.193:45549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
2026-09-29T11:38:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3065 ms)
[----------] 6 tests from EncryptionPolicyTest (21550 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:05 dist-test-slave-c0l9 krb5kdc[26545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:05 dist-test-slave-c0l9 krb5kdc[26545](info): set up 2 sockets
Sep 29 11:38:05 dist-test-slave-c0l9 krb5kdc[26545](info): commencing operation
krb5kdc: starting...
W20260929 11:38:06.480960 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:06 dist-test-slave-c0l9 krb5kdc[26545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:06.736832 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:07.070248 26558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:07.070608 26558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:07.070747 26558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:07.079056 26558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:07.079212 26558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:07.079273 26558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:07.079362 26558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:07.079424 26558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:07.090538 26558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:07.092031 26558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:07.093426 26558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:07.105418 26564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:07.105450 26563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:07.105700 26566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:07.106149 26558 server_base.cc:1034] running on GCE node
I20260929 11:38:07.106997 26558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:07.108646 26558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:07.109942 26558 hybrid_clock.cc:648] HybridClock initialized: now 1790681887109890 us; error 45 us; skew 500 ppm
I20260929 11:38:07.110389 26558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:07 dist-test-slave-c0l9 krb5kdc[26545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:07.113734 26558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:07.115226 26558 webserver.cc:460] Webserver started at http://127.0.0.1:37399/ using document root <none> and password file <none>
I20260929 11:38:07.115908 26558 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:07.116011 26558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:07.116282 26558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:07.119359 26558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "c43e6d30c28043b4aa8ff5d11bbad56b"
format_stamp: "Formatted at 2026-09-29 11:38:07 on dist-test-slave-c0l9"
server_key: "d06e074c62d4de8c6d3c7d961aa01859"
server_key_iv: "8f56b66a0955581e0570208ca181c2ea"
server_key_version: "encryptionkey@0"
I20260929 11:38:07.120107 26558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "c43e6d30c28043b4aa8ff5d11bbad56b"
format_stamp: "Formatted at 2026-09-29 11:38:07 on dist-test-slave-c0l9"
server_key: "d06e074c62d4de8c6d3c7d961aa01859"
server_key_iv: "8f56b66a0955581e0570208ca181c2ea"
server_key_version: "encryptionkey@0"
I20260929 11:38:07.125495 26558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 11:38:07.129153 26573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:07.130621 26558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:38:07.130837 26558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c43e6d30c28043b4aa8ff5d11bbad56b"
format_stamp: "Formatted at 2026-09-29 11:38:07 on dist-test-slave-c0l9"
server_key: "d06e074c62d4de8c6d3c7d961aa01859"
server_key_iv: "8f56b66a0955581e0570208ca181c2ea"
server_key_version: "encryptionkey@0"
I20260929 11:38:07.131044 26558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:07.147125 26558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:07.151564 26558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:07.151839 26558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:07.169307 26558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42679
I20260929 11:38:07.169329 26624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42679 every 8 connection(s)
I20260929 11:38:07.170768 26558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:07.171845 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26558
I20260929 11:38:07.172040 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:07.172693 6183 external_mini_cluster.cc:1428] Setting key fa442d6648fef4a6471657bc308a3273
I20260929 11:38:07.175161 26625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:38:07 dist-test-slave-c0l9 krb5kdc[26545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:38:07.188541 26625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: Bootstrap starting.
I20260929 11:38:07.192203 26625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:07.193581 26625 log.cc:824] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:07.197042 26625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: No bootstrap required, opened a new log
I20260929 11:38:07.197698 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:07.174555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40215 (local address 127.0.0.1:42679)
0929 11:38:07.175380 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:07.175397 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:07.175457 (+ 60us) server_negotiation.cc:407] Connection header received
0929 11:38:07.177683 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:07.177714 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:07.178256 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:07.178717 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:07.180059 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:07.182179 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:07.183523 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:07.184587 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:07.184821 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:07.189213 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:07.189223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:07.189241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:07.189284 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:07.192532 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:07.193720 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:07.193730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:07.193740 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:07.193991 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:07.194584 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:07.194589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:07.194593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:07.195273 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:07.195511 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:07.196095 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 11:38:07.196440 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":296,"threads_started":1}
I20260929 11:38:07.202178 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:07.206600 26625 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260929 11:38:07.207129 26625 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:07.207239 26625 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43e6d30c28043b4aa8ff5d11bbad56b, State: Initialized, Role: FOLLOWER
I20260929 11:38:07.207950 26625 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260929 11:38:07.208235 26625 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:07.208379 26625 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:07.208559 26625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:07.210871 26625 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260929 11:38:07.211367 26625 leader_election.cc:302] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43e6d30c28043b4aa8ff5d11bbad56b; no voters:
I20260929 11:38:07.211952 26625 leader_election.cc:288] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:07.212070 26635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:07.214310 26625 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:07.218763 26635 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [term 1 LEADER]: Becoming Leader. State: Replica: c43e6d30c28043b4aa8ff5d11bbad56b, State: Running, Role: LEADER
I20260929 11:38:07.219501 26635 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260929 11:38:07.222780 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:07.207990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.0.0.1:42679)
0929 11:38:07.208227 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:07.208234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:07.208696 (+ 462us) server_negotiation.cc:407] Connection header received
0929 11:38:07.208882 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:07.208886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:07.208948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:07.209037 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:07.210468 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:07.211293 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:07.212862 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:07.213406 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:07.213483 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:07.216063 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:07.216068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:07.216071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:07.216105 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:07.219331 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:07.220162 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:07.220167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:07.220169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:07.220312 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:07.220886 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:07.220891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:07.220894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:07.221325 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:07.221626 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:07.222257 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 11:38:07.222433 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:38:07.226217 26637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43e6d30c28043b4aa8ff5d11bbad56b. Latest consensus state: current_term: 1 leader_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260929 11:38:07.226296 26636 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43e6d30c28043b4aa8ff5d11bbad56b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260929 11:38:07.226636 26637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:07.226637 26636 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b [sys.catalog]: This master's current role is: LEADER
W20260929 11:38:07.232842 26651 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:38:07.232965 26651 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:38:07.233096 26648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:07.235317 26648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:07.244223 26648 catalog_manager.cc:1347] Generated new cluster ID: 60271f31f7884994b661065582407e78
I20260929 11:38:07.244386 26648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:07.254552 26648 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:07.256359 26648 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:07.266885 26648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43e6d30c28043b4aa8ff5d11bbad56b: Generated new TSK 0
I20260929 11:38:07.268083 26648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:07.273739 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26558
I20260929 11:38:07.275293 26620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:07.367296 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26558
W20260929 11:38:07.381803 26543 connection.cc:535] client connection to 127.0.0.1:42679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42679): Connection reset by peer (error 104)
2026-09-29T11:38:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:07 dist-test-slave-c0l9 krb5kdc[26657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:07 dist-test-slave-c0l9 krb5kdc[26657](info): set up 2 sockets
Sep 29 11:38:07 dist-test-slave-c0l9 krb5kdc[26657](info): commencing operation
krb5kdc: starting...
W20260929 11:38:08.388819 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:08 dist-test-slave-c0l9 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:08.636281 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:08.966216 26670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:08.966543 26670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:08.966686 26670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:08.973965 26670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:08.974103 26670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:08.974181 26670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:08.974264 26670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:08.974325 26670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:08.984448 26670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:08.985800 26670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:08.987108 26670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:08.998425 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:08.998487 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:08.998464 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:08.999449 26670 server_base.cc:1034] running on GCE node
I20260929 11:38:09.000257 26670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:09.001735 26670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:09.002977 26670 hybrid_clock.cc:648] HybridClock initialized: now 1790681889002921 us; error 33 us; skew 500 ppm
I20260929 11:38:09.003345 26670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:09 dist-test-slave-c0l9 krb5kdc[26657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:09.012176 26670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:09.013900 26670 webserver.cc:460] Webserver started at http://127.0.0.1:33525/ using document root <none> and password file <none>
I20260929 11:38:09.014727 26670 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:09.014866 26670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:09.015174 26670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:09.018469 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "c46eecab282449a280fa61b6ce01f772"
format_stamp: "Formatted at 2026-09-29 11:38:09 on dist-test-slave-c0l9"
server_key: "8a37909664e4845ac64ef58d88039170"
server_key_iv: "9c3169f2a59f3140827b98db5e9132c3"
server_key_version: "encryptionkey@0"
I20260929 11:38:09.019332 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "c46eecab282449a280fa61b6ce01f772"
format_stamp: "Formatted at 2026-09-29 11:38:09 on dist-test-slave-c0l9"
server_key: "8a37909664e4845ac64ef58d88039170"
server_key_iv: "9c3169f2a59f3140827b98db5e9132c3"
server_key_version: "encryptionkey@0"
I20260929 11:38:09.025467 26670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:38:09.029274 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:09.030715 26670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:09.030893 26670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c46eecab282449a280fa61b6ce01f772"
format_stamp: "Formatted at 2026-09-29 11:38:09 on dist-test-slave-c0l9"
server_key: "8a37909664e4845ac64ef58d88039170"
server_key_iv: "9c3169f2a59f3140827b98db5e9132c3"
server_key_version: "encryptionkey@0"
I20260929 11:38:09.031090 26670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:09.051885 26670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:09.055879 26670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:09.056142 26670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:09.072922 26670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40955
I20260929 11:38:09.072953 26736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40955 every 8 connection(s)
I20260929 11:38:09.074261 26670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:09.078226 26737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:09.081012 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26670
I20260929 11:38:09.081171 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:09.081570 6183 external_mini_cluster.cc:1428] Setting key a01dbabc4eceae70ec64dfa7a229bb5a
I20260929 11:38:09.091044 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772: Bootstrap starting.
I20260929 11:38:09.094951 26737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:09.096333 26737 log.cc:824] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:09.096980 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:09.083393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40939 (local address 127.0.0.1:40955)
0929 11:38:09.084167 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:09.084182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:09.084231 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:38:09.086541 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:09.086571 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:09.087126 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:09.087605 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:09.088144 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:09.090160 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:09.091654 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:09.092654 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:09.092885 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:09.094069 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:09.094080 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:09.094099 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:09.094792 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:09.094901 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:09.094940 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:09.095113 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:09.095382 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 11:38:09.095575 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":325,"threads_started":1}
I20260929 11:38:09.099996 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772: No bootstrap required, opened a new log
I20260929 11:38:09.101815 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:09.108431 26737 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } }
I20260929 11:38:09.108918 26737 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:09.109014 26737 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46eecab282449a280fa61b6ce01f772, State: Initialized, Role: FOLLOWER
I20260929 11:38:09.109624 26737 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } }
I20260929 11:38:09.109855 26737 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:09.109990 26737 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:09.110159 26737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:09.112224 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:09.106143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40821 (local address 127.0.0.1:40955)
0929 11:38:09.106338 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:09.106345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:09.106653 (+ 308us) server_negotiation.cc:407] Connection header received
0929 11:38:09.106832 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:09.106836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:09.106905 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:09.106995 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:09.107540 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:09.108275 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:09.109768 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:09.110316 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:09.110437 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:09.111498 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:09.111503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:09.111509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:09.111525 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:09.111582 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:09.111602 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:09.111691 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:09.111900 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 11:38:09.111954 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:38:09.112840 26737 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } }
I20260929 11:38:09.113485 26737 leader_election.cc:302] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46eecab282449a280fa61b6ce01f772; no voters:
I20260929 11:38:09.114125 26737 leader_election.cc:288] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:09.114228 26747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:09.114847 26747 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [term 1 LEADER]: Becoming Leader. State: Replica: c46eecab282449a280fa61b6ce01f772, State: Running, Role: LEADER
I20260929 11:38:09.115442 26747 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } }
I20260929 11:38:09.116334 26737 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:09.118197 26749 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46eecab282449a280fa61b6ce01f772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } } }
I20260929 11:38:09.118602 26749 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:09.118916 26748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46eecab282449a280fa61b6ce01f772. Latest consensus state: current_term: 1 leader_uuid: "c46eecab282449a280fa61b6ce01f772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46eecab282449a280fa61b6ce01f772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40955 } } }
I20260929 11:38:09.119234 26748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:09.122081 26752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:09.128007 26752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:09.138305 26752 catalog_manager.cc:1347] Generated new cluster ID: 468f22835eca4167a093a93293470362
I20260929 11:38:09.138458 26752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:09.152609 26752 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:09.154058 26752 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:09.161947 26752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46eecab282449a280fa61b6ce01f772: Generated new TSK 0
I20260929 11:38:09.163125 26752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:09.171921 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26670
W20260929 11:38:09.173264 26732 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40939
W20260929 11:38:09.174077 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:09.174247 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26670
W20260929 11:38:09.188393 26655 connection.cc:535] client connection to 127.0.0.1:40955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40955): Connection reset by peer (error 104)
2026-09-29T11:38:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1806 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:09 dist-test-slave-c0l9 krb5kdc[26768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:09 dist-test-slave-c0l9 krb5kdc[26768](info): set up 2 sockets
Sep 29 11:38:09 dist-test-slave-c0l9 krb5kdc[26768](info): commencing operation
krb5kdc: starting...
W20260929 11:38:10.196995 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:10 dist-test-slave-c0l9 krb5kdc[26768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:10.443740 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:10.777055 26781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:10.777377 26781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:10.777508 26781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:10.785091 26781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:10.785226 26781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:10.785281 26781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:10.785368 26781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:10.785440 26781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:10.796044 26781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:10.797468 26781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:10.798913 26781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:10.811117 26786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:10.811157 26789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:10.811071 26787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:10.812021 26781 server_base.cc:1034] running on GCE node
I20260929 11:38:10.812980 26781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:10.814628 26781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:10.815907 26781 hybrid_clock.cc:648] HybridClock initialized: now 1790681890815847 us; error 55 us; skew 500 ppm
I20260929 11:38:10.816356 26781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:10 dist-test-slave-c0l9 krb5kdc[26768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:10.819741 26781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:10.821508 26781 webserver.cc:460] Webserver started at http://127.0.0.1:41501/ using document root <none> and password file <none>
I20260929 11:38:10.822229 26781 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:10.822335 26781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:10.822605 26781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:10.825769 26781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "1ffca74fb24f40b1b657f94ff11398f7"
format_stamp: "Formatted at 2026-09-29 11:38:10 on dist-test-slave-c0l9"
server_key: "31e1f1717cb7fb9286f1eb1b0f9f0677"
server_key_iv: "3bd3d9e9fcb7a6cc7d1fd328ad4397f3"
server_key_version: "encryptionkey@0"
I20260929 11:38:10.826574 26781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "1ffca74fb24f40b1b657f94ff11398f7"
format_stamp: "Formatted at 2026-09-29 11:38:10 on dist-test-slave-c0l9"
server_key: "31e1f1717cb7fb9286f1eb1b0f9f0677"
server_key_iv: "3bd3d9e9fcb7a6cc7d1fd328ad4397f3"
server_key_version: "encryptionkey@0"
I20260929 11:38:10.832090 26781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:38:10.836019 26796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:10.837577 26781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:38:10.837752 26781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "1ffca74fb24f40b1b657f94ff11398f7"
format_stamp: "Formatted at 2026-09-29 11:38:10 on dist-test-slave-c0l9"
server_key: "31e1f1717cb7fb9286f1eb1b0f9f0677"
server_key_iv: "3bd3d9e9fcb7a6cc7d1fd328ad4397f3"
server_key_version: "encryptionkey@0"
I20260929 11:38:10.837950 26781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:10.843742 26781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:10.848802 26781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:10.849118 26781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:10.866132 26781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39127
I20260929 11:38:10.866168 26847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39127 every 8 connection(s)
I20260929 11:38:10.867621 26781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:10.871457 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:10.877748 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26781
I20260929 11:38:10.877918 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:10.878340 6183 external_mini_cluster.cc:1428] Setting key 1bcbdb5b569dd1b8acdbc13125b52c5d
I20260929 11:38:10.882843 26848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7: Bootstrap starting.
I20260929 11:38:10.886221 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:10.880114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57123 (local address 127.0.0.1:39127)
0929 11:38:10.881016 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:10.881030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:10.881073 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:38:10.882727 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:10.882749 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:10.883116 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:10.883462 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:10.884615 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:10.884622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:10.884636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:10.884653 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:10.884761 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:10.884787 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:10.884978 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:10.885082 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 11:38:10.885264 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":409,"threads_started":1}
I20260929 11:38:10.886433 26848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:10.887760 26848 log.cc:824] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:10.890044 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:10.890683 26848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7: No bootstrap required, opened a new log
I20260929 11:38:10.897715 26848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260929 11:38:10.898073 26848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:10.898214 26848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffca74fb24f40b1b657f94ff11398f7, State: Initialized, Role: FOLLOWER
I20260929 11:38:10.898813 26848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260929 11:38:10.899050 26848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:10.899178 26848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:10.899329 26848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:10.902436 26848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260929 11:38:10.902627 26851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:10.899999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39275 (local address 127.0.0.1:39127)
0929 11:38:10.900216 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:10.900222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:10.900501 (+ 279us) server_negotiation.cc:407] Connection header received
0929 11:38:10.900766 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:10.900772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:10.900829 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:10.900917 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:10.901923 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:10.901928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:10.901931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:10.901946 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:10.902005 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:10.902021 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:10.902123 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:10.902237 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 11:38:10.902286 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:38:10.903028 26848 leader_election.cc:302] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ffca74fb24f40b1b657f94ff11398f7; no voters:
I20260929 11:38:10.903630 26848 leader_election.cc:288] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:10.903766 26858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:10.904647 26858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffca74fb24f40b1b657f94ff11398f7, State: Running, Role: LEADER
I20260929 11:38:10.905169 26858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260929 11:38:10.905687 26848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:10.907655 26860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ffca74fb24f40b1b657f94ff11398f7. Latest consensus state: current_term: 1 leader_uuid: "1ffca74fb24f40b1b657f94ff11398f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } } }
I20260929 11:38:10.908082 26860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:10.908690 26859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ffca74fb24f40b1b657f94ff11398f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ffca74fb24f40b1b657f94ff11398f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } } }
I20260929 11:38:10.909006 26859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:10.910943 26864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:10.916648 26864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:10.927450 26864 catalog_manager.cc:1347] Generated new cluster ID: 3f771623242e42b0b36cb3a9343582ee
I20260929 11:38:10.927567 26864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:10.951612 26864 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:10.952055 26864 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:10.959698 26864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ffca74fb24f40b1b657f94ff11398f7: Generated new TSK 0
I20260929 11:38:10.960570 26864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:10.970520 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26781
W20260929 11:38:10.971763 26843 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57123
W20260929 11:38:10.972569 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26781: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:10.972774 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26781
2026-09-29T11:38:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:11 dist-test-slave-c0l9 krb5kdc[26880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:11 dist-test-slave-c0l9 krb5kdc[26880](info): set up 2 sockets
Sep 29 11:38:11 dist-test-slave-c0l9 krb5kdc[26880](info): commencing operation
krb5kdc: starting...
W20260929 11:38:12.000869 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:12.247722 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:12.581903 26893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:12.582221 26893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:12.582343 26893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:12.589790 26893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:12.589921 26893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:12.589998 26893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:12.590085 26893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:12.590143 26893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:12.600238 26893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:12.601629 26893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:12.602916 26893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:12.614995 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:12.615064 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:12.615211 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:12.615912 26893 server_base.cc:1034] running on GCE node
I20260929 11:38:12.616830 26893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:12.618368 26893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:12.619575 26893 hybrid_clock.cc:648] HybridClock initialized: now 1790681892619526 us; error 49 us; skew 500 ppm
I20260929 11:38:12.619916 26893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:12.623034 26893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:12.624521 26893 webserver.cc:460] Webserver started at http://127.0.0.1:44551/ using document root <none> and password file <none>
I20260929 11:38:12.625247 26893 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:12.625370 26893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:12.625658 26893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:12.628971 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "6a826a6422d54f45ac22cc78f9b2c829"
format_stamp: "Formatted at 2026-09-29 11:38:12 on dist-test-slave-c0l9"
server_key: "a2dc55aed0c559653ab57f1d1002a377"
server_key_iv: "289ee3c2a7451573e0879d6eebe91de1"
server_key_version: "encryptionkey@0"
I20260929 11:38:12.629739 26893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "6a826a6422d54f45ac22cc78f9b2c829"
format_stamp: "Formatted at 2026-09-29 11:38:12 on dist-test-slave-c0l9"
server_key: "a2dc55aed0c559653ab57f1d1002a377"
server_key_iv: "289ee3c2a7451573e0879d6eebe91de1"
server_key_version: "encryptionkey@0"
I20260929 11:38:12.634917 26893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:38:12.638435 26908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:12.639802 26893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:38:12.639976 26893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "6a826a6422d54f45ac22cc78f9b2c829"
format_stamp: "Formatted at 2026-09-29 11:38:12 on dist-test-slave-c0l9"
server_key: "a2dc55aed0c559653ab57f1d1002a377"
server_key_iv: "289ee3c2a7451573e0879d6eebe91de1"
server_key_version: "encryptionkey@0"
I20260929 11:38:12.640175 26893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:12.653126 26893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:12.657589 26893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:12.657874 26893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:12.675410 26893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43351
I20260929 11:38:12.675429 26959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43351 every 8 connection(s)
I20260929 11:38:12.676858 26893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:12.680704 26960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:12.681967 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 26893
I20260929 11:38:12.682129 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:12.682572 6183 external_mini_cluster.cc:1428] Setting key 88f67f84faef734f109f55373a28895d
I20260929 11:38:12.694165 26960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829: Bootstrap starting.
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:38:12.698057 26960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:12.699617 26960 log.cc:824] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:12.703862 26960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829: No bootstrap required, opened a new log
I20260929 11:38:12.707556 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:12.684293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42045 (local address 127.0.0.1:43351)
0929 11:38:12.685109 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:12.685120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:12.685164 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:38:12.686775 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:12.686797 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:12.687208 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:12.687526 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:12.689265 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:12.690551 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:12.692319 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:12.693019 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:12.693161 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:12.698162 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:12.698176 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:12.698198 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:12.698254 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:12.702627 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:12.703517 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:12.703527 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:12.703545 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:12.703773 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:12.704219 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:12.704225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:12.704228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:12.705093 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:12.705427 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:12.705951 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 11:38:12.706462 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":347,"threads_started":1}
I20260929 11:38:12.711666 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:12.712781 26960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } }
I20260929 11:38:12.713173 26960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:12.713267 26960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a826a6422d54f45ac22cc78f9b2c829, State: Initialized, Role: FOLLOWER
I20260929 11:38:12.713873 26960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } }
I20260929 11:38:12.714108 26960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:12.714251 26960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:12.714429 26960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:12.717100 26960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } }
I20260929 11:38:12.717815 26960 leader_election.cc:302] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a826a6422d54f45ac22cc78f9b2c829; no voters:
I20260929 11:38:12.718500 26960 leader_election.cc:288] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:12.718587 26970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:12.720968 26970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [term 1 LEADER]: Becoming Leader. State: Replica: 6a826a6422d54f45ac22cc78f9b2c829, State: Running, Role: LEADER
I20260929 11:38:12.721112 26960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:12.721580 26970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } }
I20260929 11:38:12.726557 26971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a826a6422d54f45ac22cc78f9b2c829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } } }
I20260929 11:38:12.727015 26971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:12.729710 26972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a826a6422d54f45ac22cc78f9b2c829. Latest consensus state: current_term: 1 leader_uuid: "6a826a6422d54f45ac22cc78f9b2c829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a826a6422d54f45ac22cc78f9b2c829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43351 } } }
I20260929 11:38:12.730103 26972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:12.734227 26979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:12.735595 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:12.716129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56219 (local address 127.0.0.1:43351)
0929 11:38:12.716333 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:12.716338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:12.716708 (+ 370us) server_negotiation.cc:407] Connection header received
0929 11:38:12.716952 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:12.716957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:12.717024 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:12.717126 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:12.719087 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:12.720140 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:12.722250 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:12.722968 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:12.723127 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:12.726820 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:12.726828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:12.726832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:12.726887 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:12.731454 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:12.732632 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:12.732639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:12.732643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:12.732857 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:12.733399 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:12.733405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:12.733407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:12.733906 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:12.734894 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:12.735064 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 11:38:12.735333 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:38:12.740167 26979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:12.741125 26987 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:38:12.749835 26979 catalog_manager.cc:1347] Generated new cluster ID: 65f7d25c86874c988e4d3cf68532daff
I20260929 11:38:12.749954 26979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:12.769899 26979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:12.771229 26979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:12.777089 26979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a826a6422d54f45ac22cc78f9b2c829: Generated new TSK 0
I20260929 11:38:12.778092 26979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:12.792577 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 26893
I20260929 11:38:12.793874 26955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:12.886078 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 26893
2026-09-29T11:38:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26992](info): set up 2 sockets
Sep 29 11:38:12 dist-test-slave-c0l9 krb5kdc[26992](info): commencing operation
krb5kdc: starting...
W20260929 11:38:13.908901 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:14 dist-test-slave-c0l9 krb5kdc[26992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:14.155675 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:14.486122 27005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:14.486454 27005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:14.486594 27005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:14.494087 27005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:14.494226 27005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:14.494277 27005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:14.494364 27005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:14.494415 27005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:14.504626 27005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:14.506062 27005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:14.507486 27005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:14.519805 27013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:14.519790 27011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:14.519747 27010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:14.520411 27005 server_base.cc:1034] running on GCE node
I20260929 11:38:14.521385 27005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:14.522869 27005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:14.524067 27005 hybrid_clock.cc:648] HybridClock initialized: now 1790681894524024 us; error 39 us; skew 500 ppm
I20260929 11:38:14.524430 27005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:14 dist-test-slave-c0l9 krb5kdc[26992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:14.530735 27005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:14.532240 27005 webserver.cc:460] Webserver started at http://127.0.0.1:39453/ using document root <none> and password file <none>
I20260929 11:38:14.533015 27005 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:14.533124 27005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:14.533391 27005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:14.536566 27005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "ae4647a5a1ed4ec883b0284606d3af39"
format_stamp: "Formatted at 2026-09-29 11:38:14 on dist-test-slave-c0l9"
server_key: "77e02264129240ea1e4a369f46cea4bd"
server_key_iv: "762fa979e0910d1e43a6043468d29152"
server_key_version: "encryptionkey@0"
I20260929 11:38:14.537462 27005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "ae4647a5a1ed4ec883b0284606d3af39"
format_stamp: "Formatted at 2026-09-29 11:38:14 on dist-test-slave-c0l9"
server_key: "77e02264129240ea1e4a369f46cea4bd"
server_key_iv: "762fa979e0910d1e43a6043468d29152"
server_key_version: "encryptionkey@0"
I20260929 11:38:14.543275 27005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 11:38:14.546993 27020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:14.548712 27005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:38:14.548899 27005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "ae4647a5a1ed4ec883b0284606d3af39"
format_stamp: "Formatted at 2026-09-29 11:38:14 on dist-test-slave-c0l9"
server_key: "77e02264129240ea1e4a369f46cea4bd"
server_key_iv: "762fa979e0910d1e43a6043468d29152"
server_key_version: "encryptionkey@0"
I20260929 11:38:14.549103 27005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:14.584945 27005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:14.589378 27005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:14.589655 27005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:14.606331 27005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35421
I20260929 11:38:14.606333 27071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35421 every 8 connection(s)
I20260929 11:38:14.607656 27005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:14.609874 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27005
I20260929 11:38:14.610055 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:14.610464 6183 external_mini_cluster.cc:1428] Setting key 5dca084e38b86ac034601cb56ce48e97
I20260929 11:38:14.611567 27072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:14.624953 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:14.612211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56099 (local address 127.0.0.1:35421)
0929 11:38:14.613024 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:14.613039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:14.613089 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:38:14.614843 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:14.614871 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:14.615379 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:14.615802 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:14.616291 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:14.618299 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:14.619756 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:14.620765 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:14.620938 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:14.622138 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:14.622148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:14.622167 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:14.622880 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:14.622972 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:14.623013 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:14.623113 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:14.623376 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:38:14.623669 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":321,"threads_started":1}
I20260929 11:38:14.625717 27072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39: Bootstrap starting.
I20260929 11:38:14.628872 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:14.629266 27072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:14.630417 27072 log.cc:824] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:14.633627 27072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39: No bootstrap required, opened a new log
I20260929 11:38:14.640856 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:14.633775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48881 (local address 127.0.0.1:35421)
0929 11:38:14.633960 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:14.633967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:14.634364 (+ 397us) server_negotiation.cc:407] Connection header received
0929 11:38:14.634568 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:14.634572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:14.634636 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:14.634715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:14.635364 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:14.636158 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:14.638185 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:14.638694 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:14.638810 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:14.640012 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:14.640019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:14.640023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:14.640043 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:14.640088 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:14.640108 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:14.640179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:14.640384 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 11:38:14.640477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:38:14.643271 27072 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260929 11:38:14.643651 27072 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:14.643749 27072 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4647a5a1ed4ec883b0284606d3af39, State: Initialized, Role: FOLLOWER
I20260929 11:38:14.644323 27072 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260929 11:38:14.644536 27072 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:14.644685 27072 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:14.644847 27072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:14.647187 27072 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260929 11:38:14.647703 27072 leader_election.cc:302] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4647a5a1ed4ec883b0284606d3af39; no voters:
I20260929 11:38:14.648267 27072 leader_election.cc:288] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:14.648438 27082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:14.649221 27082 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [term 1 LEADER]: Becoming Leader. State: Replica: ae4647a5a1ed4ec883b0284606d3af39, State: Running, Role: LEADER
I20260929 11:38:14.649796 27082 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260929 11:38:14.650568 27072 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:14.653045 27084 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4647a5a1ed4ec883b0284606d3af39. Latest consensus state: current_term: 1 leader_uuid: "ae4647a5a1ed4ec883b0284606d3af39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } } }
I20260929 11:38:14.653514 27084 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:14.653453 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4647a5a1ed4ec883b0284606d3af39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4647a5a1ed4ec883b0284606d3af39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } } }
I20260929 11:38:14.653761 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:14.657260 27089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:14.662137 27089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:14.671506 27089 catalog_manager.cc:1347] Generated new cluster ID: 72badcd814b043f88f832c1b6ef76341
I20260929 11:38:14.671633 27089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:14.691308 27089 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:14.692813 27089 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:14.704032 27089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4647a5a1ed4ec883b0284606d3af39: Generated new TSK 0
I20260929 11:38:14.705027 27089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:14.716426 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27005
W20260929 11:38:14.717669 27067 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56099
W20260929 11:38:14.718346 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27005: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:14.718485 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27005
2026-09-29T11:38:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1832 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:14 dist-test-slave-c0l9 krb5kdc[27103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:14 dist-test-slave-c0l9 krb5kdc[27103](info): set up 2 sockets
Sep 29 11:38:14 dist-test-slave-c0l9 krb5kdc[27103](info): commencing operation
krb5kdc: starting...
W20260929 11:38:15.729007 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:15 dist-test-slave-c0l9 krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:15.984771 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:16.313082 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:16.313390 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:16.313514 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:16.320844 27116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:16.320967 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:16.321028 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:16.321113 27116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:16.321187 27116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:16.331188 27116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:16.332477 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:16.333807 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:16.345551 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:16.345518 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:16.345585 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:16.346494 27116 server_base.cc:1034] running on GCE node
I20260929 11:38:16.347235 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:16.348630 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:16.349869 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790681896349823 us; error 37 us; skew 500 ppm
I20260929 11:38:16.350199 27116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:16 dist-test-slave-c0l9 krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:16.353240 27116 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:16.354796 27116 webserver.cc:460] Webserver started at http://127.0.0.1:36701/ using document root <none> and password file <none>
I20260929 11:38:16.355511 27116 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:16.355643 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:16.355947 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:16.359155 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "1012c94204b744c98a18c36e32acaa67"
format_stamp: "Formatted at 2026-09-29 11:38:16 on dist-test-slave-c0l9"
server_key: "749b4365d9adc0f073b6792c8dd72f11"
server_key_iv: "f590a319041af76d3e86dd0713677521"
server_key_version: "encryptionkey@0"
I20260929 11:38:16.359985 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "1012c94204b744c98a18c36e32acaa67"
format_stamp: "Formatted at 2026-09-29 11:38:16 on dist-test-slave-c0l9"
server_key: "749b4365d9adc0f073b6792c8dd72f11"
server_key_iv: "f590a319041af76d3e86dd0713677521"
server_key_version: "encryptionkey@0"
I20260929 11:38:16.365309 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:38:16.368706 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:16.370041 27116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:16.370183 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "1012c94204b744c98a18c36e32acaa67"
format_stamp: "Formatted at 2026-09-29 11:38:16 on dist-test-slave-c0l9"
server_key: "749b4365d9adc0f073b6792c8dd72f11"
server_key_iv: "f590a319041af76d3e86dd0713677521"
server_key_version: "encryptionkey@0"
I20260929 11:38:16.370344 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:16.375975 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:16.379913 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:16.380157 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:16.397027 27116 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46777
I20260929 11:38:16.397073 27182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46777 every 8 connection(s)
I20260929 11:38:16.398383 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:16.398967 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27116
I20260929 11:38:16.399132 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:16.399705 6183 external_mini_cluster.cc:1428] Setting key 5eb1694ff387eada599c5306a7fd053b
I20260929 11:38:16.402868 27183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:16.408461 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:16.401467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50331 (local address 127.0.0.1:46777)
0929 11:38:16.402234 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:16.402249 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:16.402298 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:38:16.404534 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:16.404596 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:16.405046 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:16.405487 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:16.406471 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:16.406480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:16.406496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:16.406512 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:16.406612 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:16.406639 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:16.406850 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:16.406998 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 11:38:16.407206 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":297,"threads_started":1}
I20260929 11:38:16.412284 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:16.415755 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67: Bootstrap starting.
I20260929 11:38:16.419064 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:16.416389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53599 (local address 127.0.0.1:46777)
0929 11:38:16.416605 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:16.416612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:16.417008 (+ 396us) server_negotiation.cc:407] Connection header received
0929 11:38:16.417243 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:16.417247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:16.417314 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:16.417435 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:16.418336 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:16.418341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:16.418344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:16.418359 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:16.418401 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:16.418416 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:16.418533 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:16.418711 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 11:38:16.418773 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:38:16.419706 27183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:16.420882 27183 log.cc:824] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:16.423604 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67: No bootstrap required, opened a new log
I20260929 11:38:16.430908 27183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260929 11:38:16.431351 27183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:16.431473 27183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1012c94204b744c98a18c36e32acaa67, State: Initialized, Role: FOLLOWER
I20260929 11:38:16.432152 27183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260929 11:38:16.432367 27183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:16.432495 27183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:16.432650 27183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:16.434919 27183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260929 11:38:16.435446 27183 leader_election.cc:302] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1012c94204b744c98a18c36e32acaa67; no voters:
I20260929 11:38:16.436108 27183 leader_election.cc:288] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:16.436374 27193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:16.437096 27193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [term 1 LEADER]: Becoming Leader. State: Replica: 1012c94204b744c98a18c36e32acaa67, State: Running, Role: LEADER
I20260929 11:38:16.437664 27193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260929 11:38:16.438294 27183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:16.440359 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1012c94204b744c98a18c36e32acaa67. Latest consensus state: current_term: 1 leader_uuid: "1012c94204b744c98a18c36e32acaa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260929 11:38:16.440897 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:16.440862 27194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1012c94204b744c98a18c36e32acaa67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1012c94204b744c98a18c36e32acaa67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260929 11:38:16.441140 27194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:16.443758 27202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:16.448000 27202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:16.456854 27202 catalog_manager.cc:1347] Generated new cluster ID: f304183e123a4b079e0b604562c2d2f5
I20260929 11:38:16.457005 27202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:16.487113 27202 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:16.487527 27202 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:16.502461 27202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1012c94204b744c98a18c36e32acaa67: Generated new TSK 0
I20260929 11:38:16.503310 27202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:16.516125 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27116
W20260929 11:38:16.517553 27178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50331
W20260929 11:38:16.518244 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27116: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:16.518371 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27116
2026-09-29T11:38:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1800 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:16 dist-test-slave-c0l9 krb5kdc[27214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:16 dist-test-slave-c0l9 krb5kdc[27214](info): set up 2 sockets
Sep 29 11:38:16 dist-test-slave-c0l9 krb5kdc[27214](info): commencing operation
krb5kdc: starting...
W20260929 11:38:17.548923 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:17 dist-test-slave-c0l9 krb5kdc[27214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:17.796746 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:18.134954 27227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:18.135291 27227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:18.135449 27227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:18.143059 27227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:18.143205 27227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:18.143270 27227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:18.143378 27227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:18.143442 27227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:18.153884 27227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:18.155238 27227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:18.156594 27227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:18.168767 27232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:18.168769 27233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:18.168783 27235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:18.169588 27227 server_base.cc:1034] running on GCE node
I20260929 11:38:18.170397 27227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:18.171762 27227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:18.172971 27227 hybrid_clock.cc:648] HybridClock initialized: now 1790681898172922 us; error 42 us; skew 500 ppm
I20260929 11:38:18.173326 27227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:18 dist-test-slave-c0l9 krb5kdc[27214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:18.176258 27227 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:18.177696 27227 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260929 11:38:18.178411 27227 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:18.178534 27227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:18.178829 27227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:18.182003 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "ec14ab48f9ee413a83c3550f8df3b59b"
format_stamp: "Formatted at 2026-09-29 11:38:18 on dist-test-slave-c0l9"
server_key: "dca69eb6f4e0f06e37d4cf89fbcde6ab"
server_key_iv: "bd509ec93556cd50e80844556be51e27"
server_key_version: "encryptionkey@0"
I20260929 11:38:18.182776 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "ec14ab48f9ee413a83c3550f8df3b59b"
format_stamp: "Formatted at 2026-09-29 11:38:18 on dist-test-slave-c0l9"
server_key: "dca69eb6f4e0f06e37d4cf89fbcde6ab"
server_key_iv: "bd509ec93556cd50e80844556be51e27"
server_key_version: "encryptionkey@0"
I20260929 11:38:18.188076 27227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:38:18.191808 27242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:18.193261 27227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:38:18.193432 27227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "ec14ab48f9ee413a83c3550f8df3b59b"
format_stamp: "Formatted at 2026-09-29 11:38:18 on dist-test-slave-c0l9"
server_key: "dca69eb6f4e0f06e37d4cf89fbcde6ab"
server_key_iv: "bd509ec93556cd50e80844556be51e27"
server_key_version: "encryptionkey@0"
I20260929 11:38:18.193610 27227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:18.220403 27227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:18.224529 27227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:18.224810 27227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:18.242197 27227 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45817
I20260929 11:38:18.242216 27293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45817 every 8 connection(s)
I20260929 11:38:18.243515 27227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:18.247152 27294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:18.251461 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27227
I20260929 11:38:18.251641 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:18.252161 6183 external_mini_cluster.cc:1428] Setting key f68cb49cdecada441dfee5a3d1e7cc81
I20260929 11:38:18.259608 27294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b: Bootstrap starting.
I20260929 11:38:18.263437 27294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:18.264647 27294 log.cc:824] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:18.267727 27294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b: No bootstrap required, opened a new log
Sep 29 11:38:18 dist-test-slave-c0l9 krb5kdc[27214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:38:18.278043 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:18.254144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:33899 (local address 127.0.0.1:45817)
0929 11:38:18.255056 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:18.255072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:18.255420 (+ 348us) server_negotiation.cc:407] Connection header received
0929 11:38:18.257994 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:18.258026 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:18.258590 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:18.259067 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:18.260467 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:18.262535 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:18.264013 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:18.265122 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:18.265346 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:18.269505 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:18.269515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:18.269535 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:18.269581 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:18.273547 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:18.274358 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:18.274366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:18.274375 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:18.274585 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:18.274988 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:18.274993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:18.274996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:18.275715 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:18.276028 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:18.276328 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 11:38:18.276710 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260929 11:38:18.277772 27294 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 11:38:18.278205 27294 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:18.278308 27294 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec14ab48f9ee413a83c3550f8df3b59b, State: Initialized, Role: FOLLOWER
I20260929 11:38:18.278889 27294 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 11:38:18.279098 27294 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:18.279182 27294 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:18.279306 27294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:18.281711 27294 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 11:38:18.281862 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:18.282241 27294 leader_election.cc:302] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec14ab48f9ee413a83c3550f8df3b59b; no voters:
I20260929 11:38:18.282831 27294 leader_election.cc:288] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:18.282923 27299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:18.284044 27299 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [term 1 LEADER]: Becoming Leader. State: Replica: ec14ab48f9ee413a83c3550f8df3b59b, State: Running, Role: LEADER
I20260929 11:38:18.284550 27299 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 11:38:18.284904 27294 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:18.286983 27301 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec14ab48f9ee413a83c3550f8df3b59b. Latest consensus state: current_term: 1 leader_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260929 11:38:18.287350 27301 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:18.287873 27300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec14ab48f9ee413a83c3550f8df3b59b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260929 11:38:18.288151 27300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:18.295722 27306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:18.301818 27306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:18.328953 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:18.298724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:55641 (local address 127.0.0.1:45817)
0929 11:38:18.298917 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:18.298924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:18.309442 (+ 10518us) server_negotiation.cc:407] Connection header received
0929 11:38:18.309681 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:18.309689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:18.309789 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:18.309869 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:18.312261 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:18.313533 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:18.315599 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:18.316282 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:18.316448 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:18.319918 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:18.319926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:18.319931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:18.319972 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:18.323982 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:18.325151 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:18.325157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:18.325166 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:18.325336 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:18.326040 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:18.326046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:18.326050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:18.326566 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:18.326845 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:18.328283 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0929 11:38:18.328498 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:38:18.329875 27306 catalog_manager.cc:1347] Generated new cluster ID: 79c5f4d7bd694a06b0be462a1c0c561c
I20260929 11:38:18.329991 27306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:18.345963 27306 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:18.347934 27306 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:18.356724 27306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec14ab48f9ee413a83c3550f8df3b59b: Generated new TSK 0
I20260929 11:38:18.357661 27306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:18.371781 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27227
I20260929 11:38:18.373039 27289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:18.467054 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27227
W20260929 11:38:18.481177 27212 connection.cc:535] client connection to 127.0.0.1:45817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45817): Connection reset by peer (error 104)
2026-09-29T11:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:18 dist-test-slave-c0l9 krb5kdc[27327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:18 dist-test-slave-c0l9 krb5kdc[27327](info): set up 2 sockets
Sep 29 11:38:18 dist-test-slave-c0l9 krb5kdc[27327](info): commencing operation
krb5kdc: starting...
W20260929 11:38:19.484903 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:19 dist-test-slave-c0l9 krb5kdc[27327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:19.737102 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:20.070420 27340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:20.070735 27340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:20.070856 27340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:20.078186 27340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:20.078310 27340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:20.078365 27340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:20.078454 27340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:20.078517 27340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:20.089300 27340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:20.090852 27340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:20.092276 27340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:20.104885 27345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:20.104864 27346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:20.104910 27348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:20.105674 27340 server_base.cc:1034] running on GCE node
I20260929 11:38:20.106524 27340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:20.108052 27340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:20.109290 27340 hybrid_clock.cc:648] HybridClock initialized: now 1790681900109233 us; error 47 us; skew 500 ppm
I20260929 11:38:20.109669 27340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:20 dist-test-slave-c0l9 krb5kdc[27327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:20.118165 27340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:20.119796 27340 webserver.cc:460] Webserver started at http://127.0.0.1:45581/ using document root <none> and password file <none>
I20260929 11:38:20.120740 27340 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:20.120886 27340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:20.121188 27340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:20.124277 27340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "0df1920468d2456d994a2367b741c8cb"
format_stamp: "Formatted at 2026-09-29 11:38:20 on dist-test-slave-c0l9"
server_key: "76cce995080256a1c4074994aaf68b49"
server_key_iv: "8dd8256e04104e2fcf97b1ca03cd1f2c"
server_key_version: "encryptionkey@0"
I20260929 11:38:20.125062 27340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "0df1920468d2456d994a2367b741c8cb"
format_stamp: "Formatted at 2026-09-29 11:38:20 on dist-test-slave-c0l9"
server_key: "76cce995080256a1c4074994aaf68b49"
server_key_iv: "8dd8256e04104e2fcf97b1ca03cd1f2c"
server_key_version: "encryptionkey@0"
I20260929 11:38:20.130373 27340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:38:20.134069 27355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:20.135450 27340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:20.135615 27340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "0df1920468d2456d994a2367b741c8cb"
format_stamp: "Formatted at 2026-09-29 11:38:20 on dist-test-slave-c0l9"
server_key: "76cce995080256a1c4074994aaf68b49"
server_key_iv: "8dd8256e04104e2fcf97b1ca03cd1f2c"
server_key_version: "encryptionkey@0"
I20260929 11:38:20.135802 27340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:20.159725 27340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:20.163952 27340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:20.164350 27340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:20.181126 27340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42307
I20260929 11:38:20.181169 27406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42307 every 8 connection(s)
I20260929 11:38:20.182498 27340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:20.186115 27407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:20.192361 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27340
I20260929 11:38:20.192543 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:20.192996 6183 external_mini_cluster.cc:1428] Setting key 5ce6c3bf22287c8bee2d63be80dca163
I20260929 11:38:20.196306 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb: Bootstrap starting.
I20260929 11:38:20.199574 27407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:20.200769 27407 log.cc:824] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:20.203791 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb: No bootstrap required, opened a new log
I20260929 11:38:20.206378 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:20.194755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:48973 (local address 127.0.0.1:42307)
0929 11:38:20.195465 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:20.195478 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:20.195520 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:38:20.197234 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:20.197257 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:20.197652 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:20.197994 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:20.198561 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:20.199949 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:20.201770 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:20.202528 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:20.202656 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:20.203863 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:20.203871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:20.203907 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:20.204543 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:20.204627 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:20.204655 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:20.204812 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:20.205141 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 11:38:20.205325 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260929 11:38:20.210438 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:20.213393 27407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260929 11:38:20.213902 27407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:20.214020 27407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df1920468d2456d994a2367b741c8cb, State: Initialized, Role: FOLLOWER
I20260929 11:38:20.214758 27407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260929 11:38:20.215047 27407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:20.215193 27407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:20.215384 27407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:20.218470 27407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260929 11:38:20.219072 27407 leader_election.cc:302] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df1920468d2456d994a2367b741c8cb; no voters:
I20260929 11:38:20.219698 27407 leader_election.cc:288] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:20.219843 27417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:20.220446 27417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [term 1 LEADER]: Becoming Leader. State: Replica: 0df1920468d2456d994a2367b741c8cb, State: Running, Role: LEADER
I20260929 11:38:20.221176 27417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260929 11:38:20.223903 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:20.215864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:38715 (local address 127.0.0.1:42307)
0929 11:38:20.216062 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:20.216068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:20.216532 (+ 464us) server_negotiation.cc:407] Connection header received
0929 11:38:20.216742 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:20.216748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:20.216839 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:20.216986 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:20.217567 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:20.218609 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:20.220359 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:20.221071 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:20.221200 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:20.222771 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:20.222778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:20.222783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:20.222811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:20.222860 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:20.222886 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:20.223013 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:20.223505 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 11:38:20.223565 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:38:20.224226 27407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:20.224313 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df1920468d2456d994a2367b741c8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } } }
I20260929 11:38:20.224784 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:20.226657 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df1920468d2456d994a2367b741c8cb. Latest consensus state: current_term: 1 leader_uuid: "0df1920468d2456d994a2367b741c8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df1920468d2456d994a2367b741c8cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } } }
I20260929 11:38:20.226999 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:20.227517 27421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:20.232573 27421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:20.241712 27421 catalog_manager.cc:1347] Generated new cluster ID: fa79debcefd343a29fc8df7dca88f2d6
I20260929 11:38:20.241827 27421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:20.255942 27421 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:20.257405 27421 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:20.264453 27421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df1920468d2456d994a2367b741c8cb: Generated new TSK 0
I20260929 11:38:20.265568 27421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:20.280346 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27340
W20260929 11:38:20.281541 27402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:48973
W20260929 11:38:20.282404 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:20.282552 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27340
W20260929 11:38:20.296459 27325 connection.cc:535] client connection to 127.0.0.1:42307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42307): Connection reset by peer (error 104)
2026-09-29T11:38:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1815 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:20 dist-test-slave-c0l9 krb5kdc[27438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:20 dist-test-slave-c0l9 krb5kdc[27438](info): set up 2 sockets
Sep 29 11:38:20 dist-test-slave-c0l9 krb5kdc[27438](info): commencing operation
krb5kdc: starting...
W20260929 11:38:21.300874 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:21 dist-test-slave-c0l9 krb5kdc[27438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:21.559501 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:21.903545 27451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:21.903838 27451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:21.903972 27451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:21.911217 27451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:21.911331 27451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:21.911401 27451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:21.911492 27451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:21.911548 27451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:21.921597 27451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:21.922900 27451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:21.924187 27451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:21.935714 27456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:21.935743 27457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:21.935680 27459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:21.936117 27451 server_base.cc:1034] running on GCE node
I20260929 11:38:21.937233 27451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:21.938769 27451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:21.939981 27451 hybrid_clock.cc:648] HybridClock initialized: now 1790681901939920 us; error 48 us; skew 500 ppm
I20260929 11:38:21.940348 27451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:21 dist-test-slave-c0l9 krb5kdc[27438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:21.943400 27451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:21.944912 27451 webserver.cc:460] Webserver started at http://127.0.0.1:45687/ using document root <none> and password file <none>
I20260929 11:38:21.945626 27451 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:21.945737 27451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:21.946039 27451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:21.949101 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "ddaef9afc6de4a45835659b2857c2385"
format_stamp: "Formatted at 2026-09-29 11:38:21 on dist-test-slave-c0l9"
server_key: "b0a105791d481b3990e9e9ea69b33ebf"
server_key_iv: "33ed986cb1c0097af56a00b678e37531"
server_key_version: "encryptionkey@0"
I20260929 11:38:21.949836 27451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "ddaef9afc6de4a45835659b2857c2385"
format_stamp: "Formatted at 2026-09-29 11:38:21 on dist-test-slave-c0l9"
server_key: "b0a105791d481b3990e9e9ea69b33ebf"
server_key_iv: "33ed986cb1c0097af56a00b678e37531"
server_key_version: "encryptionkey@0"
I20260929 11:38:21.954909 27451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:38:21.958462 27466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:21.959847 27451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:38:21.960013 27451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "ddaef9afc6de4a45835659b2857c2385"
format_stamp: "Formatted at 2026-09-29 11:38:21 on dist-test-slave-c0l9"
server_key: "b0a105791d481b3990e9e9ea69b33ebf"
server_key_iv: "33ed986cb1c0097af56a00b678e37531"
server_key_version: "encryptionkey@0"
I20260929 11:38:21.960201 27451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:21.966114 27451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:21.970287 27451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:21.970569 27451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:21.988234 27451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42889
I20260929 11:38:21.988281 27517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42889 every 8 connection(s)
I20260929 11:38:21.989714 27451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:21.993726 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27451
I20260929 11:38:21.993767 27518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:21.993914 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:21.994441 6183 external_mini_cluster.cc:1428] Setting key 9a8b2f5337623113bac3c3c043991495
I20260929 11:38:22.002470 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:21.996284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:49907 (local address 127.0.0.1:42889)
0929 11:38:21.997273 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:21.997284 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:21.997332 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:38:21.998908 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:21.998937 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:21.999319 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:21.999718 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:22.000870 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:22.000878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:22.000898 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:22.000914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:22.000982 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:22.001007 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:22.001207 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:22.001321 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 11:38:22.001510 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":453,"threads_started":1}
I20260929 11:38:22.006242 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:22.007558 27518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385: Bootstrap starting.
I20260929 11:38:22.010716 27518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:22.011989 27518 log.cc:824] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:22.012949 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:22.010467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:59381 (local address 127.0.0.1:42889)
0929 11:38:22.010654 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:22.010659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:22.011020 (+ 361us) server_negotiation.cc:407] Connection header received
0929 11:38:22.011192 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:22.011196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:22.011247 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:22.011333 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:22.012267 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:22.012271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:22.012274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:22.012288 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:22.012325 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:22.012340 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:22.012421 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:22.012571 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 11:38:22.012647 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:38:22.014961 27518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385: No bootstrap required, opened a new log
I20260929 11:38:22.022884 27518 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260929 11:38:22.023275 27518 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:22.023384 27518 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaef9afc6de4a45835659b2857c2385, State: Initialized, Role: FOLLOWER
I20260929 11:38:22.023953 27518 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260929 11:38:22.024190 27518 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:22.024317 27518 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:22.024466 27518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:22.026942 27518 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260929 11:38:22.027415 27518 leader_election.cc:302] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddaef9afc6de4a45835659b2857c2385; no voters:
I20260929 11:38:22.027933 27518 leader_election.cc:288] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:22.028074 27528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:22.028848 27528 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [term 1 LEADER]: Becoming Leader. State: Replica: ddaef9afc6de4a45835659b2857c2385, State: Running, Role: LEADER
I20260929 11:38:22.029412 27528 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } }
I20260929 11:38:22.029958 27518 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:22.032294 27529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddaef9afc6de4a45835659b2857c2385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260929 11:38:22.032740 27529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:22.033145 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddaef9afc6de4a45835659b2857c2385. Latest consensus state: current_term: 1 leader_uuid: "ddaef9afc6de4a45835659b2857c2385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddaef9afc6de4a45835659b2857c2385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42889 } } }
I20260929 11:38:22.033455 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:22.035727 27537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:22.040113 27537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:22.048709 27537 catalog_manager.cc:1347] Generated new cluster ID: 5aa8cea469844242ba69fc789ee36b71
I20260929 11:38:22.048820 27537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:22.070712 27537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:22.071199 27537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:22.085109 27537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddaef9afc6de4a45835659b2857c2385: Generated new TSK 0
I20260929 11:38:22.086007 27537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:22.094097 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27451
W20260929 11:38:22.095296 27513 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:49907
W20260929 11:38:22.096092 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27451: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:22.096230 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27451
2026-09-29T11:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1813 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:22 dist-test-slave-c0l9 krb5kdc[27549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:22 dist-test-slave-c0l9 krb5kdc[27549](info): set up 2 sockets
Sep 29 11:38:22 dist-test-slave-c0l9 krb5kdc[27549](info): commencing operation
krb5kdc: starting...
W20260929 11:38:23.112776 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:23 dist-test-slave-c0l9 krb5kdc[27549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:23.359477 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:23.685649 27562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:23.686015 27562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:23.686167 27562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:23.693670 27562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:23.693809 27562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:23.693889 27562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:23.693979 27562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:23.694038 27562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:23.704381 27562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:23.705740 27562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:23.707033 27562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:23.718698 27567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:23.718667 27570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:23.718735 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:23.719420 27562 server_base.cc:1034] running on GCE node
I20260929 11:38:23.720252 27562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:23.721755 27562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:23.722973 27562 hybrid_clock.cc:648] HybridClock initialized: now 1790681903722917 us; error 44 us; skew 500 ppm
I20260929 11:38:23.723335 27562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:23 dist-test-slave-c0l9 krb5kdc[27549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:23.726382 27562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:23.727806 27562 webserver.cc:460] Webserver started at http://127.0.0.1:42301/ using document root <none> and password file <none>
I20260929 11:38:23.728493 27562 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:23.728608 27562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:23.728914 27562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:23.731858 27562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "38294f195dda405586b6f1fb5d4728b5"
format_stamp: "Formatted at 2026-09-29 11:38:23 on dist-test-slave-c0l9"
server_key: "c85bf67a8dde934f5fd083e48240e81e"
server_key_iv: "b40a203c2e201e6dfeb9330a8d2abfff"
server_key_version: "encryptionkey@0"
I20260929 11:38:23.732613 27562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "38294f195dda405586b6f1fb5d4728b5"
format_stamp: "Formatted at 2026-09-29 11:38:23 on dist-test-slave-c0l9"
server_key: "c85bf67a8dde934f5fd083e48240e81e"
server_key_iv: "b40a203c2e201e6dfeb9330a8d2abfff"
server_key_version: "encryptionkey@0"
I20260929 11:38:23.737886 27562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:38:23.741490 27577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:23.742884 27562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:23.743062 27562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "38294f195dda405586b6f1fb5d4728b5"
format_stamp: "Formatted at 2026-09-29 11:38:23 on dist-test-slave-c0l9"
server_key: "c85bf67a8dde934f5fd083e48240e81e"
server_key_iv: "b40a203c2e201e6dfeb9330a8d2abfff"
server_key_version: "encryptionkey@0"
I20260929 11:38:23.743253 27562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:23.777725 27562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:23.782431 27562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:23.782724 27562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:23.800705 27562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42547
I20260929 11:38:23.800750 27628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42547 every 8 connection(s)
I20260929 11:38:23.802181 27562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:23.804564 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27562
I20260929 11:38:23.804797 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:23.805347 6183 external_mini_cluster.cc:1428] Setting key e271dc50a7f4b96575faa9cea86ac234
I20260929 11:38:23.806766 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:38:23 dist-test-slave-c0l9 krb5kdc[27549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:38:23.821117 27629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5: Bootstrap starting.
I20260929 11:38:23.824465 27629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:23.825767 27629 log.cc:824] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:23.829046 27629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5: No bootstrap required, opened a new log
I20260929 11:38:23.829463 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:23.807220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:58519 (local address 127.0.0.1:42547)
0929 11:38:23.808000 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:23.808013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:23.808055 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:38:23.809736 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:23.809759 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:23.810174 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:23.810546 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:23.812332 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:23.813839 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:23.815693 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:23.816400 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:23.816576 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:23.821799 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:23.821807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:23.821824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:23.821875 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:23.824908 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:23.825893 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:23.825900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:23.825906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:23.826104 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:23.826642 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:23.826647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:23.826650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:23.827318 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:23.827555 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:23.828093 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 11:38:23.828393 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":342,"threads_started":1}
I20260929 11:38:23.833179 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:23.838359 27629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260929 11:38:23.838804 27629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:23.838910 27629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38294f195dda405586b6f1fb5d4728b5, State: Initialized, Role: FOLLOWER
I20260929 11:38:23.839567 27629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260929 11:38:23.839848 27629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:23.839985 27629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:23.840163 27629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:23.842989 27629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260929 11:38:23.843577 27629 leader_election.cc:302] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38294f195dda405586b6f1fb5d4728b5; no voters:
I20260929 11:38:23.844272 27629 leader_election.cc:288] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:23.844373 27639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:23.844970 27639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [term 1 LEADER]: Becoming Leader. State: Replica: 38294f195dda405586b6f1fb5d4728b5, State: Running, Role: LEADER
I20260929 11:38:23.845567 27639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } }
I20260929 11:38:23.848272 27640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38294f195dda405586b6f1fb5d4728b5. Latest consensus state: current_term: 1 leader_uuid: "38294f195dda405586b6f1fb5d4728b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260929 11:38:23.848893 27640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:23.849290 27629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:23.851219 27641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38294f195dda405586b6f1fb5d4728b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38294f195dda405586b6f1fb5d4728b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42547 } } }
I20260929 11:38:23.851532 27641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:23.853132 27643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:23.855404 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:23.838395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:47865 (local address 127.0.0.1:42547)
0929 11:38:23.838578 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:23.838584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:23.839137 (+ 553us) server_negotiation.cc:407] Connection header received
0929 11:38:23.839349 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:23.839355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:23.839434 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:23.839523 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:23.841387 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:23.842167 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:23.843875 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:23.844320 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:23.844444 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:23.847258 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:23.847264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:23.847269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:23.847312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:23.851128 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:23.852279 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:23.852285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:23.852288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:23.852494 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:23.853141 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:23.853145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:23.853148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:23.853588 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:23.854626 (+ 1038us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:23.854826 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 11:38:23.855022 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:38:23.859045 27643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:23.867525 27643 catalog_manager.cc:1347] Generated new cluster ID: 1eb0f3f6217249f4a879cbed89415cd9
I20260929 11:38:23.867640 27643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:23.890529 27643 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:23.892501 27643 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:23.905260 27643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38294f195dda405586b6f1fb5d4728b5: Generated new TSK 0
I20260929 11:38:23.906387 27643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:23.914181 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27562
I20260929 11:38:23.915490 27624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:24.006872 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27562
W20260929 11:38:24.021205 27547 connection.cc:535] client connection to 127.0.0.1:42547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42547): Connection reset by peer (error 104)
2026-09-29T11:38:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:24 dist-test-slave-c0l9 krb5kdc[27661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:24 dist-test-slave-c0l9 krb5kdc[27661](info): set up 2 sockets
Sep 29 11:38:24 dist-test-slave-c0l9 krb5kdc[27661](info): commencing operation
krb5kdc: starting...
W20260929 11:38:25.040953 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:25 dist-test-slave-c0l9 krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:25.290040 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:25.629226 27674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:25.629566 27674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:25.629727 27674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:25.637195 27674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:25.637348 27674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:25.637441 27674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:25.637524 27674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:25.637593 27674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:25.648063 27674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:25.649510 27674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:25.650889 27674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:25.663118 27680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:25.663153 27679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:25.663738 27682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:25.664326 27674 server_base.cc:1034] running on GCE node
I20260929 11:38:25.665180 27674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:25.666692 27674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:25.667925 27674 hybrid_clock.cc:648] HybridClock initialized: now 1790681905667872 us; error 42 us; skew 500 ppm
I20260929 11:38:25.668323 27674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:25 dist-test-slave-c0l9 krb5kdc[27661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:25.676944 27674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:25.678557 27674 webserver.cc:460] Webserver started at http://127.0.0.1:36817/ using document root <none> and password file <none>
I20260929 11:38:25.679282 27674 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:25.679394 27674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:25.679653 27674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:25.682789 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "4fc5b0fb626f4e0f8207537478a9938d"
format_stamp: "Formatted at 2026-09-29 11:38:25 on dist-test-slave-c0l9"
server_key: "7fad656f14c90f29e3a7742443120d7f"
server_key_iv: "260831c05f5cb54d0287bf53a6dad25e"
server_key_version: "encryptionkey@0"
I20260929 11:38:25.683576 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "4fc5b0fb626f4e0f8207537478a9938d"
format_stamp: "Formatted at 2026-09-29 11:38:25 on dist-test-slave-c0l9"
server_key: "7fad656f14c90f29e3a7742443120d7f"
server_key_iv: "260831c05f5cb54d0287bf53a6dad25e"
server_key_version: "encryptionkey@0"
I20260929 11:38:25.688953 27674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 11:38:25.692484 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:25.693913 27674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:38:25.694065 27674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "4fc5b0fb626f4e0f8207537478a9938d"
format_stamp: "Formatted at 2026-09-29 11:38:25 on dist-test-slave-c0l9"
server_key: "7fad656f14c90f29e3a7742443120d7f"
server_key_iv: "260831c05f5cb54d0287bf53a6dad25e"
server_key_version: "encryptionkey@0"
I20260929 11:38:25.694239 27674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:25.729164 27674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:25.733383 27674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:25.733654 27674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:25.752604 27674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34339
I20260929 11:38:25.752629 27740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34339 every 8 connection(s)
I20260929 11:38:25.754140 27674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:25.757840 27741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:25.758778 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27674
I20260929 11:38:25.758963 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:25.759431 6183 external_mini_cluster.cc:1428] Setting key 55874f453ee32503c98d5e0e69382755
I20260929 11:38:25.772562 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d: Bootstrap starting.
I20260929 11:38:25.774611 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:25.761386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:57145 (local address 127.0.0.1:34339)
0929 11:38:25.762202 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:25.762218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:25.762273 (+ 55us) server_negotiation.cc:407] Connection header received
0929 11:38:25.764207 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:25.764239 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:25.764850 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:25.765314 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:25.765836 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:25.767661 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:25.769163 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:25.770278 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:25.770472 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:25.771690 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:25.771700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:25.771720 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:25.772441 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:25.772540 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:25.772577 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:25.772765 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:25.773118 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 11:38:25.773291 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":290,"threads_started":1}
I20260929 11:38:25.776197 27741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:25.777349 27741 log.cc:824] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:25.778380 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:25.780455 27741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d: No bootstrap required, opened a new log
I20260929 11:38:25.790159 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:25.783453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:50929 (local address 127.0.0.1:34339)
0929 11:38:25.783635 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:25.783640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:25.783969 (+ 329us) server_negotiation.cc:407] Connection header received
0929 11:38:25.784126 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:25.784131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:25.784216 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:25.784321 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:25.784952 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:25.786096 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:25.787503 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:25.788181 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:25.788311 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:25.789380 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:25.789386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:25.789391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:25.789415 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:25.789460 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:25.789481 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:25.789591 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:25.789765 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 11:38:25.789820 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:38:25.789862 27741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260929 11:38:25.790356 27741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:25.790472 27741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc5b0fb626f4e0f8207537478a9938d, State: Initialized, Role: FOLLOWER
I20260929 11:38:25.791054 27741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260929 11:38:25.791291 27741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:25.791441 27741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:25.791635 27741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:25.794094 27741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260929 11:38:25.794592 27741 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc5b0fb626f4e0f8207537478a9938d; no voters:
I20260929 11:38:25.795162 27741 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:25.795300 27751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:25.796000 27751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [term 1 LEADER]: Becoming Leader. State: Replica: 4fc5b0fb626f4e0f8207537478a9938d, State: Running, Role: LEADER
I20260929 11:38:25.796521 27751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260929 11:38:25.797269 27741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:25.799566 27752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc5b0fb626f4e0f8207537478a9938d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260929 11:38:25.800009 27752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:25.800460 27753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc5b0fb626f4e0f8207537478a9938d. Latest consensus state: current_term: 1 leader_uuid: "4fc5b0fb626f4e0f8207537478a9938d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc5b0fb626f4e0f8207537478a9938d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260929 11:38:25.800854 27753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:25.801755 27760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:25.806836 27760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:25.815627 27760 catalog_manager.cc:1347] Generated new cluster ID: 3889a202cd424cfc94f064e1f95e2e75
I20260929 11:38:25.815752 27760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:25.830044 27760 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:25.831545 27760 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:25.848050 27760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc5b0fb626f4e0f8207537478a9938d: Generated new TSK 0
I20260929 11:38:25.849277 27760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:25.855787 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27674
W20260929 11:38:25.857577 27736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:57145
W20260929 11:38:25.858361 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:25.858495 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27674
W20260929 11:38:25.873199 27659 connection.cc:535] client connection to 127.0.0.1:34339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34339): Connection reset by peer (error 104)
2026-09-29T11:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:25 dist-test-slave-c0l9 krb5kdc[27772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:25 dist-test-slave-c0l9 krb5kdc[27772](info): set up 2 sockets
Sep 29 11:38:25 dist-test-slave-c0l9 krb5kdc[27772](info): commencing operation
krb5kdc: starting...
W20260929 11:38:26.876935 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:27 dist-test-slave-c0l9 krb5kdc[27772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:38:27.119494 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:27.450670 27785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:27.450961 27785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:27.451061 27785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:27.458206 27785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:27.458324 27785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:27.458369 27785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:27.458427 27785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:27.458470 27785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:27.468756 27785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:27.470059 27785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:27.471328 27785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:27.482914 27790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:27.482952 27791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:27.482971 27793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:27.483423 27785 server_base.cc:1034] running on GCE node
I20260929 11:38:27.484447 27785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:27.485873 27785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:27.487087 27785 hybrid_clock.cc:648] HybridClock initialized: now 1790681907487040 us; error 39 us; skew 500 ppm
I20260929 11:38:27.487421 27785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:27 dist-test-slave-c0l9 krb5kdc[27772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:27.490479 27785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:38:27.491891 27785 webserver.cc:460] Webserver started at http://127.0.0.1:41383/ using document root <none> and password file <none>
I20260929 11:38:27.492527 27785 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:27.492640 27785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:27.492995 27785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:27.495863 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "c42a3ead79f046dc8dadae3258b52092"
format_stamp: "Formatted at 2026-09-29 11:38:27 on dist-test-slave-c0l9"
server_key: "4ef2e2beb2a4eccb7f4823042c4d0b3f"
server_key_iv: "276eab98df088de187f300872babc013"
server_key_version: "encryptionkey@0"
I20260929 11:38:27.496585 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "c42a3ead79f046dc8dadae3258b52092"
format_stamp: "Formatted at 2026-09-29 11:38:27 on dist-test-slave-c0l9"
server_key: "4ef2e2beb2a4eccb7f4823042c4d0b3f"
server_key_iv: "276eab98df088de187f300872babc013"
server_key_version: "encryptionkey@0"
I20260929 11:38:27.501467 27785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:38:27.504757 27800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:27.505985 27785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:27.506127 27785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "c42a3ead79f046dc8dadae3258b52092"
format_stamp: "Formatted at 2026-09-29 11:38:27 on dist-test-slave-c0l9"
server_key: "4ef2e2beb2a4eccb7f4823042c4d0b3f"
server_key_iv: "276eab98df088de187f300872babc013"
server_key_version: "encryptionkey@0"
I20260929 11:38:27.506299 27785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:27.511812 27785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:27.515727 27785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:27.515993 27785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:27.532925 27785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36927
I20260929 11:38:27.532960 27851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36927 every 8 connection(s)
I20260929 11:38:27.534319 27785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:27.538013 27852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:27.543954 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27785
I20260929 11:38:27.544188 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:27.544714 6183 external_mini_cluster.cc:1428] Setting key 64d8c894988ec6e15562092e06672115
I20260929 11:38:27.549014 27852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092: Bootstrap starting.
I20260929 11:38:27.552522 27852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:27.552561 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:27.546607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:56419 (local address 127.0.0.1:36927)
0929 11:38:27.547395 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:27.547408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:27.547449 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:38:27.549270 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:27.549294 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:27.549631 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:27.549954 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:27.551026 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:27.551032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:27.551048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:27.551065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:27.551122 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:27.551144 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:27.551302 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:27.551433 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 11:38:27.551610 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":343,"threads_started":1}
I20260929 11:38:27.553857 27852 log.cc:824] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:27.556527 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:27.556972 27852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092: No bootstrap required, opened a new log
I20260929 11:38:27.562942 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:27.560704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:57181 (local address 127.0.0.1:36927)
0929 11:38:27.560872 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:27.560878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:27.561195 (+ 317us) server_negotiation.cc:407] Connection header received
0929 11:38:27.561368 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:27.561372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:27.561419 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:27.561503 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:27.562333 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:27.562339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:27.562344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:27.562361 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:27.562398 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:27.562415 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:27.562533 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:27.562604 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 11:38:27.562664 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 11:38:27.565237 27852 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 11:38:27.565585 27852 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:27.565680 27852 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a3ead79f046dc8dadae3258b52092, State: Initialized, Role: FOLLOWER
I20260929 11:38:27.566213 27852 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 11:38:27.566443 27852 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:27.566573 27852 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:27.566736 27852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:27.569284 27852 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 11:38:27.569742 27852 leader_election.cc:302] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42a3ead79f046dc8dadae3258b52092; no voters:
I20260929 11:38:27.570307 27852 leader_election.cc:288] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:27.570432 27862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:27.571023 27862 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [term 1 LEADER]: Becoming Leader. State: Replica: c42a3ead79f046dc8dadae3258b52092, State: Running, Role: LEADER
I20260929 11:38:27.571496 27862 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } }
I20260929 11:38:27.572232 27852 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:27.574146 27864 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42a3ead79f046dc8dadae3258b52092. Latest consensus state: current_term: 1 leader_uuid: "c42a3ead79f046dc8dadae3258b52092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260929 11:38:27.574539 27864 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:27.575942 27863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42a3ead79f046dc8dadae3258b52092" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a3ead79f046dc8dadae3258b52092" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36927 } } }
I20260929 11:38:27.576184 27863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:27.576771 27869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:27.581809 27869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:27.590982 27869 catalog_manager.cc:1347] Generated new cluster ID: b1cb6dbb5ab84999a6d2b6e6ade41c5a
I20260929 11:38:27.591086 27869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:27.603061 27869 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:27.603557 27869 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:27.612812 27869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42a3ead79f046dc8dadae3258b52092: Generated new TSK 0
I20260929 11:38:27.613927 27869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:27.626055 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27785
W20260929 11:38:27.627183 27847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:56419
W20260929 11:38:27.627936 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:27.628101 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27785
2026-09-29T11:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:27 dist-test-slave-c0l9 krb5kdc[27883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:27 dist-test-slave-c0l9 krb5kdc[27883](info): set up 2 sockets
Sep 29 11:38:27 dist-test-slave-c0l9 krb5kdc[27883](info): commencing operation
krb5kdc: starting...
W20260929 11:38:28.644914 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:28 dist-test-slave-c0l9 krb5kdc[27883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:28.887758 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42661
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:38811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:29.206483 27896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:29.206808 27896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:29.206948 27896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:29.214275 27896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:29.214396 27896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:29.214452 27896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:29.214537 27896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:29.214587 27896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:29.224608 27896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:38811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:42661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:29.225943 27896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:29.227226 27896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:29.238555 27901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:29.238629 27904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:29.238592 27902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:29.239480 27896 server_base.cc:1034] running on GCE node
I20260929 11:38:29.240285 27896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:29.241758 27896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:29.242954 27896 hybrid_clock.cc:648] HybridClock initialized: now 1790681909242913 us; error 33 us; skew 500 ppm
I20260929 11:38:29.243287 27896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:29 dist-test-slave-c0l9 krb5kdc[27883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681909, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:29.246377 27896 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:29.247845 27896 webserver.cc:460] Webserver started at http://127.6.9.254:40037/ using document root <none> and password file <none>
I20260929 11:38:29.248502 27896 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:29.248616 27896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:29.248930 27896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:29.251833 27896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "280fafb312eb4325adaee68979a75a95"
format_stamp: "Formatted at 2026-09-29 11:38:29 on dist-test-slave-c0l9"
server_key: "8575d45fa3aefe8e963ad3d7c7aa3e75"
server_key_iv: "8a883757145587e87baddc6b984a583b"
server_key_version: "encryptionkey@0"
I20260929 11:38:29.252565 27896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "280fafb312eb4325adaee68979a75a95"
format_stamp: "Formatted at 2026-09-29 11:38:29 on dist-test-slave-c0l9"
server_key: "8575d45fa3aefe8e963ad3d7c7aa3e75"
server_key_iv: "8a883757145587e87baddc6b984a583b"
server_key_version: "encryptionkey@0"
I20260929 11:38:29.257795 27896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:38:29.261539 27911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:29.262981 27896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:29.263163 27896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "280fafb312eb4325adaee68979a75a95"
format_stamp: "Formatted at 2026-09-29 11:38:29 on dist-test-slave-c0l9"
server_key: "8575d45fa3aefe8e963ad3d7c7aa3e75"
server_key_iv: "8a883757145587e87baddc6b984a583b"
server_key_version: "encryptionkey@0"
I20260929 11:38:29.263357 27896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:29.287122 27896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:29.291720 27896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:29.292002 27896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:29.309325 27896 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:42661
I20260929 11:38:29.309350 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:42661 every 8 connection(s)
I20260929 11:38:29.310681 27896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:29.314399 27963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:29.320818 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 27896
I20260929 11:38:29.320996 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:29.321417 6183 external_mini_cluster.cc:1428] Setting key af5ffe758984d4a4bc10f9fded80145f
I20260929 11:38:29.325250 27963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95: Bootstrap starting.
I20260929 11:38:29.328895 27963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:29.329993 27963 log.cc:824] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:29.333227 27963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95: No bootstrap required, opened a new log
Sep 29 11:38:29 dist-test-slave-c0l9 krb5kdc[27883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:29.341176 27963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } }
I20260929 11:38:29.341517 27963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:29.341619 27963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280fafb312eb4325adaee68979a75a95, State: Initialized, Role: FOLLOWER
I20260929 11:38:29.342214 27963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } }
I20260929 11:38:29.342450 27963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:29.342594 27963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:29.342813 27963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:29.343901 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:29.323266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34829 (local address 127.6.9.254:42661)
0929 11:38:29.324006 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:29.324017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:29.324060 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:38:29.325842 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:29.325866 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:29.326261 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:29.326590 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:29.328393 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:29.329786 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:29.331292 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:29.331947 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:29.332069 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:29.336654 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:29.336704 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:29.336723 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:29.336766 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:29.339752 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:29.340516 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:29.340522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:29.340529 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:29.340727 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:29.341164 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:29.341169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:29.341173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:29.341762 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:29.342060 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:29.342547 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 11:38:29.342856 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260929 11:38:29.345309 27963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } }
I20260929 11:38:29.345779 27963 leader_election.cc:302] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280fafb312eb4325adaee68979a75a95; no voters:
I20260929 11:38:29.347077 27963 leader_election.cc:288] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:29.347209 27968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:29.347915 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:29.348435 27968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [term 1 LEADER]: Becoming Leader. State: Replica: 280fafb312eb4325adaee68979a75a95, State: Running, Role: LEADER
I20260929 11:38:29.348954 27968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } }
I20260929 11:38:29.349192 27963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:29.351322 27969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280fafb312eb4325adaee68979a75a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } } }
I20260929 11:38:29.352156 27969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:29.353684 27974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:29.353523 27970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280fafb312eb4325adaee68979a75a95. Latest consensus state: current_term: 1 leader_uuid: "280fafb312eb4325adaee68979a75a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280fafb312eb4325adaee68979a75a95" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42661 } } }
I20260929 11:38:29.353922 27970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:29.358690 27974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:29.385815 27974 catalog_manager.cc:1347] Generated new cluster ID: 25488e6c021f41b3b01a56584a55d52b
I20260929 11:38:29.385948 27974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:29.404352 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:29.374109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32953 (local address 127.6.9.254:42661)
0929 11:38:29.374311 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:29.374327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:29.384864 (+ 10537us) server_negotiation.cc:407] Connection header received
0929 11:38:29.385421 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:29.385428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:29.385539 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:29.385617 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:29.387860 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:29.389216 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:29.391258 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:29.391978 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:29.392091 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:29.395699 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:29.395706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:29.395712 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:29.395761 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:29.399622 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:29.400695 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:29.400701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:29.400706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:29.400886 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:29.401518 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:29.401525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:29.401529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:29.401982 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:29.402273 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:29.403670 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0929 11:38:29.403898 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71}
I20260929 11:38:29.409449 27974 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:29.411065 27974 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:29.418210 27974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280fafb312eb4325adaee68979a75a95: Generated new TSK 0
I20260929 11:38:29.419066 27974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:29.428629 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 27896
I20260929 11:38:29.430624 27958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:29.525200 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 27896
W20260929 11:38:29.539423 27881 connection.cc:535] client connection to 127.6.9.254:42661 recv error: Network error: failed to read from TLS socket (remote: 127.6.9.254:42661): Connection reset by peer (error 104)
2026-09-29T11:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:29 dist-test-slave-c0l9 krb5kdc[27995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:29 dist-test-slave-c0l9 krb5kdc[27995](info): set up 2 sockets
Sep 29 11:38:29 dist-test-slave-c0l9 krb5kdc[27995](info): commencing operation
krb5kdc: starting...
W20260929 11:38:30.544960 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:30 dist-test-slave-c0l9 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:30.792738 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:35397
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:35397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:31.116855 28008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:31.117201 28008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:31.117340 28008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:31.124773 28008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:31.124908 28008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:31.124984 28008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:31.125066 28008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:31.125125 28008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:31.135231 28008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:35397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:35397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:31.136777 28008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:31.138064 28008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:31.149101 28016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:31.149258 28014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:31.149205 28013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:31.150146 28008 server_base.cc:1034] running on GCE node
I20260929 11:38:31.150906 28008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:31.152333 28008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:31.153564 28008 hybrid_clock.cc:648] HybridClock initialized: now 1790681911153504 us; error 51 us; skew 500 ppm
I20260929 11:38:31.153950 28008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:31 dist-test-slave-c0l9 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681911, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:31.160386 28008 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:31.162091 28008 webserver.cc:460] Webserver started at http://127.6.9.254:43585/ using document root <none> and password file <none>
I20260929 11:38:31.162853 28008 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:31.162988 28008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:31.163293 28008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:31.166821 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "20a7847cae394f60a8a102c69740b52c"
format_stamp: "Formatted at 2026-09-29 11:38:31 on dist-test-slave-c0l9"
server_key: "fcfa6a3c240f860b85c9e2b64f904622"
server_key_iv: "f0f28eeef1424edb2f81d5dd6e8a309b"
server_key_version: "encryptionkey@0"
I20260929 11:38:31.167702 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "20a7847cae394f60a8a102c69740b52c"
format_stamp: "Formatted at 2026-09-29 11:38:31 on dist-test-slave-c0l9"
server_key: "fcfa6a3c240f860b85c9e2b64f904622"
server_key_iv: "f0f28eeef1424edb2f81d5dd6e8a309b"
server_key_version: "encryptionkey@0"
I20260929 11:38:31.173527 28008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 11:38:31.177476 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:31.178957 28008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:31.179154 28008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "20a7847cae394f60a8a102c69740b52c"
format_stamp: "Formatted at 2026-09-29 11:38:31 on dist-test-slave-c0l9"
server_key: "fcfa6a3c240f860b85c9e2b64f904622"
server_key_iv: "f0f28eeef1424edb2f81d5dd6e8a309b"
server_key_version: "encryptionkey@0"
I20260929 11:38:31.179356 28008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:31.206178 28008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:31.210650 28008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:31.210944 28008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:31.228367 28008 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:35397
I20260929 11:38:31.228386 28074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:35397 every 8 connection(s)
I20260929 11:38:31.229846 28008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:31.233820 28075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:31.237063 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28008
I20260929 11:38:31.237259 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:31.237798 6183 external_mini_cluster.cc:1428] Setting key d6d040160e25ac21afe3c89c65ba6c08
I20260929 11:38:31.246428 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c: Bootstrap starting.
I20260929 11:38:31.250321 28075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:31.251571 28075 log.cc:824] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:31.252053 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:31.239820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38425 (local address 127.6.9.254:35397)
0929 11:38:31.240646 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:31.240682 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:31.240735 (+ 53us) server_negotiation.cc:407] Connection header received
0929 11:38:31.242452 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:31.242476 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:31.242939 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:31.243288 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:31.243888 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:31.245282 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:31.247203 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:31.247907 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:31.248060 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:31.249624 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:31.249632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:31.249649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:31.250218 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:31.250303 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:31.250333 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:31.250494 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:31.250766 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:38:31.250975 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":384,"threads_started":1}
I20260929 11:38:31.254670 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c: No bootstrap required, opened a new log
I20260929 11:38:31.255847 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:31.262811 28075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } }
I20260929 11:38:31.263243 28075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:31.263337 28075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a7847cae394f60a8a102c69740b52c, State: Initialized, Role: FOLLOWER
I20260929 11:38:31.264003 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } }
I20260929 11:38:31.264320 28075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:31.264475 28075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:31.264703 28075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:31.266386 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:31.259976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53589 (local address 127.6.9.254:35397)
0929 11:38:31.260130 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:31.260134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:31.260597 (+ 463us) server_negotiation.cc:407] Connection header received
0929 11:38:31.260815 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:31.260819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:31.260888 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:31.260998 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:31.261500 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:31.262268 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:31.263691 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:31.264334 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:31.264482 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:31.265630 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:31.265636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:31.265644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:31.265671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:31.265739 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:31.265763 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:31.265877 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:31.266006 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 11:38:31.266064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:38:31.267823 28075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } }
I20260929 11:38:31.268306 28075 leader_election.cc:302] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20a7847cae394f60a8a102c69740b52c; no voters:
I20260929 11:38:31.268882 28075 leader_election.cc:288] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:31.269354 28085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:31.270287 28085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [term 1 LEADER]: Becoming Leader. State: Replica: 20a7847cae394f60a8a102c69740b52c, State: Running, Role: LEADER
I20260929 11:38:31.270814 28085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } }
I20260929 11:38:31.270954 28075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:31.273296 28087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20a7847cae394f60a8a102c69740b52c. Latest consensus state: current_term: 1 leader_uuid: "20a7847cae394f60a8a102c69740b52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } } }
I20260929 11:38:31.273669 28087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:31.274165 28086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20a7847cae394f60a8a102c69740b52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a7847cae394f60a8a102c69740b52c" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 35397 } } }
I20260929 11:38:31.274466 28086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:31.274664 28094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:31.280678 28094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:31.289961 28094 catalog_manager.cc:1347] Generated new cluster ID: 08ba1d9af1d24c3d92866c5843b848f1
I20260929 11:38:31.290119 28094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:31.318434 28094 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:31.319989 28094 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:31.330991 28094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20a7847cae394f60a8a102c69740b52c: Generated new TSK 0
I20260929 11:38:31.331907 28094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:31.350621 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28008
W20260929 11:38:31.351828 28070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38425
W20260929 11:38:31.352596 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:31.352775 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28008
W20260929 11:38:31.367177 27993 connection.cc:535] client connection to 127.6.9.254:35397 recv error: Network error: failed to read from TLS socket (remote: 127.6.9.254:35397): Connection reset by peer (error 104)
2026-09-29T11:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:31 dist-test-slave-c0l9 krb5kdc[28106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:31 dist-test-slave-c0l9 krb5kdc[28106](info): set up 2 sockets
Sep 29 11:38:31 dist-test-slave-c0l9 krb5kdc[28106](info): commencing operation
krb5kdc: starting...
W20260929 11:38:32.388854 6183 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:32 dist-test-slave-c0l9 krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:32.636322 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33033
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:32.962082 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:32.962423 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:32.962569 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:32.969941 28119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:32.970068 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:32.970149 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:32.970240 28119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:32.970305 28119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:32.980198 28119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:32.981550 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:32.982827 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:32.994068 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:32.994084 28127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:32.994082 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:32.994961 28119 server_base.cc:1034] running on GCE node
I20260929 11:38:32.995658 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:32.997032 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:32.998225 28119 hybrid_clock.cc:648] HybridClock initialized: now 1790681912998178 us; error 35 us; skew 500 ppm
I20260929 11:38:32.998544 28119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:32 dist-test-slave-c0l9 krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681912, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:33.001421 28119 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:33.002735 28119 webserver.cc:460] Webserver started at http://127.6.9.254:41009/ using document root <none> and password file <none>
I20260929 11:38:33.003350 28119 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:33.003451 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:33.003737 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:33.006582 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "8068ff4e08b74570a4cc674a00b072b4"
format_stamp: "Formatted at 2026-09-29 11:38:32 on dist-test-slave-c0l9"
server_key: "c211584219a18fe732fc99b554f0fad7"
server_key_iv: "e53fa8f3ad507486f9f979d0c719b071"
server_key_version: "encryptionkey@0"
I20260929 11:38:33.007293 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "8068ff4e08b74570a4cc674a00b072b4"
format_stamp: "Formatted at 2026-09-29 11:38:32 on dist-test-slave-c0l9"
server_key: "c211584219a18fe732fc99b554f0fad7"
server_key_iv: "e53fa8f3ad507486f9f979d0c719b071"
server_key_version: "encryptionkey@0"
I20260929 11:38:33.012072 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 11:38:33.015375 28134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:33.016561 28119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:38:33.016779 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "8068ff4e08b74570a4cc674a00b072b4"
format_stamp: "Formatted at 2026-09-29 11:38:32 on dist-test-slave-c0l9"
server_key: "c211584219a18fe732fc99b554f0fad7"
server_key_iv: "e53fa8f3ad507486f9f979d0c719b071"
server_key_version: "encryptionkey@0"
I20260929 11:38:33.016963 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:33.022142 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:33.025761 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:33.026026 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:33.042495 28119 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33033
I20260929 11:38:33.042534 28185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33033 every 8 connection(s)
I20260929 11:38:33.043845 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:33.047992 28186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:33.049669 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28119
I20260929 11:38:33.049877 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:33.050462 6183 external_mini_cluster.cc:1428] Setting key e83b7268338ba5cd18d6b39f7edad0fd
I20260929 11:38:33.057838 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:33.052264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57097 (local address 127.6.9.254:33033)
0929 11:38:33.053074 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:33.053086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:33.053131 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:38:33.054661 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:33.054686 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:33.055055 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:33.055371 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:33.056383 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:33.056389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:33.056402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:33.056415 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:33.056472 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:33.056493 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:33.056648 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:33.056783 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 11:38:33.056981 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":349,"threads_started":1}
I20260929 11:38:33.061549 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:33.061991 28186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4: Bootstrap starting.
I20260929 11:38:33.065681 28186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:33.066898 28186 log.cc:824] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:33.068919 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:33.066560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46645 (local address 127.6.9.254:33033)
0929 11:38:33.066743 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:33.066748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:33.067095 (+ 347us) server_negotiation.cc:407] Connection header received
0929 11:38:33.067272 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:33.067276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:33.067336 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:33.067411 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:33.068340 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:33.068344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:33.068348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:33.068361 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:33.068396 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:33.068410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:33.068485 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:33.068601 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 11:38:33.068641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:38:33.070089 28186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4: No bootstrap required, opened a new log
I20260929 11:38:33.077486 28186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } }
I20260929 11:38:33.077847 28186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:33.077947 28186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8068ff4e08b74570a4cc674a00b072b4, State: Initialized, Role: FOLLOWER
I20260929 11:38:33.078504 28186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } }
I20260929 11:38:33.078714 28186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:33.078845 28186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:33.078990 28186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:33.081319 28186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } }
I20260929 11:38:33.081792 28186 leader_election.cc:302] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8068ff4e08b74570a4cc674a00b072b4; no voters:
I20260929 11:38:33.082329 28186 leader_election.cc:288] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:33.082436 28196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:33.083003 28196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8068ff4e08b74570a4cc674a00b072b4, State: Running, Role: LEADER
I20260929 11:38:33.083529 28196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } }
I20260929 11:38:33.084280 28186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:33.086300 28198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8068ff4e08b74570a4cc674a00b072b4. Latest consensus state: current_term: 1 leader_uuid: "8068ff4e08b74570a4cc674a00b072b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } } }
I20260929 11:38:33.086717 28198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:33.086347 28197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8068ff4e08b74570a4cc674a00b072b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8068ff4e08b74570a4cc674a00b072b4" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33033 } } }
I20260929 11:38:33.088022 28197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:33.089102 28202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:33.093216 28202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:33.101980 28202 catalog_manager.cc:1347] Generated new cluster ID: 7d3bf0f774fb4b52992d30770e2c2f99
I20260929 11:38:33.102124 28202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:33.114398 28202 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:33.114969 28202 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:33.125350 28202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8068ff4e08b74570a4cc674a00b072b4: Generated new TSK 0
I20260929 11:38:33.126547 28202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:33.132869 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28119
W20260929 11:38:33.134097 28181 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57097
W20260929 11:38:33.134927 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28119: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:33.135067 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28119
2026-09-29T11:38:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1781 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:33 dist-test-slave-c0l9 krb5kdc[28217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:33 dist-test-slave-c0l9 krb5kdc[28217](info): set up 2 sockets
Sep 29 11:38:33 dist-test-slave-c0l9 krb5kdc[28217](info): commencing operation
krb5kdc: starting...
W20260929 11:38:34.152923 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:34 dist-test-slave-c0l9 krb5kdc[28217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:34.395984 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46289
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:46289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:34.722460 28230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:34.722784 28230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:34.722930 28230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:34.730450 28230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:34.730597 28230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:34.730675 28230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:34.730756 28230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:34.730818 28230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:34.740972 28230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:46289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:34.742319 28230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:34.743588 28230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:34.754832 28235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:34.754925 28238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:34.754868 28236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:34.755867 28230 server_base.cc:1034] running on GCE node
I20260929 11:38:34.756639 28230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:34.758064 28230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:34.759269 28230 hybrid_clock.cc:648] HybridClock initialized: now 1790681914759216 us; error 43 us; skew 500 ppm
I20260929 11:38:34.759604 28230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:34 dist-test-slave-c0l9 krb5kdc[28217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681914, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:34.762545 28230 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:34.763918 28230 webserver.cc:460] Webserver started at http://127.6.9.254:34745/ using document root <none> and password file <none>
I20260929 11:38:34.764552 28230 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:34.764649 28230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:34.764937 28230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:34.767942 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "88c1862f5aac41689dfdc3d8d794bc24"
format_stamp: "Formatted at 2026-09-29 11:38:34 on dist-test-slave-c0l9"
server_key: "d28ba8a516378072ba161727a679507c"
server_key_iv: "03a1abd60b38de18f7aef55d53120ae5"
server_key_version: "encryptionkey@0"
I20260929 11:38:34.768723 28230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "88c1862f5aac41689dfdc3d8d794bc24"
format_stamp: "Formatted at 2026-09-29 11:38:34 on dist-test-slave-c0l9"
server_key: "d28ba8a516378072ba161727a679507c"
server_key_iv: "03a1abd60b38de18f7aef55d53120ae5"
server_key_version: "encryptionkey@0"
I20260929 11:38:34.773890 28230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 11:38:34.777163 28245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:34.778512 28230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:38:34.778674 28230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "88c1862f5aac41689dfdc3d8d794bc24"
format_stamp: "Formatted at 2026-09-29 11:38:34 on dist-test-slave-c0l9"
server_key: "d28ba8a516378072ba161727a679507c"
server_key_iv: "03a1abd60b38de18f7aef55d53120ae5"
server_key_version: "encryptionkey@0"
I20260929 11:38:34.778852 28230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:34.799654 28230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:34.804164 28230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:34.804469 28230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:34.823329 28230 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:46289
I20260929 11:38:34.823351 28296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:46289 every 8 connection(s)
I20260929 11:38:34.824823 28230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:34.828522 28297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:34.831481 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28230
I20260929 11:38:34.831643 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:34.832108 6183 external_mini_cluster.cc:1428] Setting key f8a1828f3c1daa58903c3d0d8c537a56
I20260929 11:38:34.840067 28297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24: Bootstrap starting.
I20260929 11:38:34.843752 28297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:34.845091 28297 log.cc:824] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:38:34 dist-test-slave-c0l9 krb5kdc[28217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:34.848409 28297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24: No bootstrap required, opened a new log
I20260929 11:38:34.855721 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:34.833837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57057 (local address 127.6.9.254:46289)
0929 11:38:34.834572 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:34.834584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:34.834623 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:38:34.836207 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:34.836229 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:34.836634 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:34.836998 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:34.838678 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:34.839969 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:34.841765 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:34.842466 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:34.842602 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:34.847576 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:34.847587 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:34.847607 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:34.847653 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:34.851266 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:34.852182 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:34.852190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:34.852197 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:34.852381 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:34.852840 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:34.852845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:34.852847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:34.853444 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:34.853689 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:34.854175 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 11:38:34.854582 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":327,"threads_started":1}
I20260929 11:38:34.857153 28297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } }
I20260929 11:38:34.857558 28297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:34.857667 28297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c1862f5aac41689dfdc3d8d794bc24, State: Initialized, Role: FOLLOWER
I20260929 11:38:34.858254 28297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } }
I20260929 11:38:34.858530 28297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:34.858672 28297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:34.858842 28297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:34.859767 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:34.861920 28297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } }
I20260929 11:38:34.862560 28297 leader_election.cc:302] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c1862f5aac41689dfdc3d8d794bc24; no voters:
I20260929 11:38:34.863242 28297 leader_election.cc:288] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:34.863371 28303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:34.864594 28303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [term 1 LEADER]: Becoming Leader. State: Replica: 88c1862f5aac41689dfdc3d8d794bc24, State: Running, Role: LEADER
I20260929 11:38:34.865285 28303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } }
I20260929 11:38:34.865856 28297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:34.871079 28308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c1862f5aac41689dfdc3d8d794bc24. Latest consensus state: current_term: 1 leader_uuid: "88c1862f5aac41689dfdc3d8d794bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } } }
I20260929 11:38:34.871105 28306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c1862f5aac41689dfdc3d8d794bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c1862f5aac41689dfdc3d8d794bc24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 46289 } } }
I20260929 11:38:34.871544 28308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:34.871591 28306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:34.873672 28315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:34.878904 28315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:34.889220 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:34.866271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33611 (local address 127.6.9.254:46289)
0929 11:38:34.866455 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:34.866463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:34.869582 (+ 3119us) server_negotiation.cc:407] Connection header received
0929 11:38:34.872322 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:34.872332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:34.872426 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:34.872602 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:34.874565 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:34.875718 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:34.877471 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:34.878122 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:34.878260 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:34.881250 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:34.881257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:34.881262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:34.881302 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:34.885182 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:34.886074 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:34.886080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:34.886084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:34.886227 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:34.886697 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:34.886702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:34.886705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:34.887140 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:34.887383 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:34.888597 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0929 11:38:34.888846 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:38:34.895685 28315 catalog_manager.cc:1347] Generated new cluster ID: 7b8913bf6b7247379d6cd0754e52c324
I20260929 11:38:34.895843 28315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:34.919212 28315 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:34.920820 28315 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:34.932544 28315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c1862f5aac41689dfdc3d8d794bc24: Generated new TSK 0
I20260929 11:38:34.933431 28315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:34.941619 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28230
I20260929 11:38:34.942588 28292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:35.035213 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28230
2026-09-29T11:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:35 dist-test-slave-c0l9 krb5kdc[28329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:35 dist-test-slave-c0l9 krb5kdc[28329](info): set up 2 sockets
Sep 29 11:38:35 dist-test-slave-c0l9 krb5kdc[28329](info): commencing operation
krb5kdc: starting...
W20260929 11:38:36.044938 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:36 dist-test-slave-c0l9 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:36.293545 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42191
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:42191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:36.626974 28342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:36.627274 28342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:36.627395 28342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:36.634876 28342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:36.635005 28342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:36.635057 28342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:36.635144 28342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:36.635195 28342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:36.645596 28342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:42191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:36.646972 28342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:36.648607 28342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:36.660586 28350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:36.660691 28347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:36.660626 28348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:36.661406 28342 server_base.cc:1034] running on GCE node
I20260929 11:38:36.662173 28342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:36.663561 28342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:36.664773 28342 hybrid_clock.cc:648] HybridClock initialized: now 1790681916664726 us; error 42 us; skew 500 ppm
I20260929 11:38:36.665107 28342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:36 dist-test-slave-c0l9 krb5kdc[28329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:36.668094 28342 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:36.669524 28342 webserver.cc:460] Webserver started at http://127.6.9.254:45859/ using document root <none> and password file <none>
I20260929 11:38:36.670194 28342 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:36.670310 28342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:36.670608 28342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:36.673656 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "80995ea27941428091995f686fcaba12"
format_stamp: "Formatted at 2026-09-29 11:38:36 on dist-test-slave-c0l9"
server_key: "21a77a320d542f3bb01d5aad0aff48c5"
server_key_iv: "e53709e45291c3f64893b645ddcb30ec"
server_key_version: "encryptionkey@0"
I20260929 11:38:36.674396 28342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "80995ea27941428091995f686fcaba12"
format_stamp: "Formatted at 2026-09-29 11:38:36 on dist-test-slave-c0l9"
server_key: "21a77a320d542f3bb01d5aad0aff48c5"
server_key_iv: "e53709e45291c3f64893b645ddcb30ec"
server_key_version: "encryptionkey@0"
I20260929 11:38:36.679478 28342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:38:36.682852 28357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:36.684088 28342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:38:36.684247 28342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "80995ea27941428091995f686fcaba12"
format_stamp: "Formatted at 2026-09-29 11:38:36 on dist-test-slave-c0l9"
server_key: "21a77a320d542f3bb01d5aad0aff48c5"
server_key_iv: "e53709e45291c3f64893b645ddcb30ec"
server_key_version: "encryptionkey@0"
I20260929 11:38:36.684422 28342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:36.704999 28342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:36.709111 28342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:36.709373 28342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:36.726511 28342 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:42191
I20260929 11:38:36.726544 28408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:42191 every 8 connection(s)
I20260929 11:38:36.727830 28342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:36.731520 28409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:36.737934 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28342
I20260929 11:38:36.738116 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:36.738663 6183 external_mini_cluster.cc:1428] Setting key 0b8d5018277e05119a37708720d562ef
I20260929 11:38:36.742473 28409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: Bootstrap starting.
I20260929 11:38:36.746039 28409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:36.747300 28409 log.cc:824] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:36.750368 28409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: No bootstrap required, opened a new log
I20260929 11:38:36.752207 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:36.740491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46539 (local address 127.6.9.254:42191)
0929 11:38:36.741251 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:36.741262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:36.741301 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:38:36.743002 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:36.743026 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:36.743413 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:36.743729 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:36.744473 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:36.745843 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:36.747619 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:36.748636 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:36.748855 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:36.749975 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:36.749983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:36.749999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:36.750553 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:36.750627 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:36.750653 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:36.750743 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:36.750991 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 11:38:36.751257 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":372,"threads_started":1}
I20260929 11:38:36.755746 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:36.759279 28409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } }
I20260929 11:38:36.759788 28409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:36.759907 28409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80995ea27941428091995f686fcaba12, State: Initialized, Role: FOLLOWER
I20260929 11:38:36.760593 28409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } }
I20260929 11:38:36.760963 28409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:36.761122 28409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:36.761320 28409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:36.764369 28409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } }
I20260929 11:38:36.765019 28409 leader_election.cc:302] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80995ea27941428091995f686fcaba12; no voters:
I20260929 11:38:36.765727 28409 leader_election.cc:288] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:36.765852 28419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:36.767748 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:36.761027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37091 (local address 127.6.9.254:42191)
0929 11:38:36.761190 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:36.761194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:36.761621 (+ 427us) server_negotiation.cc:407] Connection header received
0929 11:38:36.761842 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:36.761845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:36.761912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:36.762001 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:36.762598 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:36.763321 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:36.765138 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:36.765661 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:36.765788 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:36.766999 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:36.767005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:36.767010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:36.767032 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:36.767077 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:36.767097 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:36.767170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:36.767339 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 11:38:36.767433 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:38:36.768151 28409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:36.772840 28419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [term 1 LEADER]: Becoming Leader. State: Replica: 80995ea27941428091995f686fcaba12, State: Running, Role: LEADER
I20260929 11:38:36.773504 28419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } }
I20260929 11:38:36.776621 28421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80995ea27941428091995f686fcaba12. Latest consensus state: current_term: 1 leader_uuid: "80995ea27941428091995f686fcaba12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } } }
I20260929 11:38:36.776634 28420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80995ea27941428091995f686fcaba12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80995ea27941428091995f686fcaba12" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42191 } } }
I20260929 11:38:36.777220 28421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:36.777231 28420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12 [sys.catalog]: This master's current role is: LEADER
W20260929 11:38:36.785796 28433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:38:36.785914 28433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:38:36.786046 28434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:36.787627 28434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:36.795730 28434 catalog_manager.cc:1347] Generated new cluster ID: 2235704ffe4943c999060af802a901ea
I20260929 11:38:36.795853 28434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:36.807030 28434 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:36.808984 28434 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:36.818006 28434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80995ea27941428091995f686fcaba12: Generated new TSK 0
I20260929 11:38:36.819128 28434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:36.825309 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28342
W20260929 11:38:36.826519 28404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46539
W20260929 11:38:36.827270 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:36.827409 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28342
2026-09-29T11:38:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1792 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:36 dist-test-slave-c0l9 krb5kdc[28440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:36 dist-test-slave-c0l9 krb5kdc[28440](info): set up 2 sockets
Sep 29 11:38:36 dist-test-slave-c0l9 krb5kdc[28440](info): commencing operation
krb5kdc: starting...
W20260929 11:38:37.840874 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:37 dist-test-slave-c0l9 krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:38.087069 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40895
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:38.409390 28453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:38.409714 28453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:38.409876 28453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:38.418542 28453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:38.418704 28453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:38.418787 28453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:38.418886 28453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:38.418962 28453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:38.429648 28453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:40895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:38.431044 28453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:38.432471 28453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:38.444763 28458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:38.444782 28459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:38.445022 28461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:38.445488 28453 server_base.cc:1034] running on GCE node
I20260929 11:38:38.446321 28453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:38.447742 28453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:38.448949 28453 hybrid_clock.cc:648] HybridClock initialized: now 1790681918448893 us; error 49 us; skew 500 ppm
I20260929 11:38:38.449307 28453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:38 dist-test-slave-c0l9 krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681918, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:38.452801 28453 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:38.454296 28453 webserver.cc:460] Webserver started at http://127.6.9.254:42147/ using document root <none> and password file <none>
I20260929 11:38:38.454977 28453 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:38.455114 28453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:38.455422 28453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:38.458565 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "fece4c8f243445d09a55951477d50c24"
format_stamp: "Formatted at 2026-09-29 11:38:38 on dist-test-slave-c0l9"
server_key: "637dcef7b0c695aa23c9fd73152e557c"
server_key_iv: "3cdbb225417c7119c42ce2e6ef88afee"
server_key_version: "encryptionkey@0"
I20260929 11:38:38.459388 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "fece4c8f243445d09a55951477d50c24"
format_stamp: "Formatted at 2026-09-29 11:38:38 on dist-test-slave-c0l9"
server_key: "637dcef7b0c695aa23c9fd73152e557c"
server_key_iv: "3cdbb225417c7119c42ce2e6ef88afee"
server_key_version: "encryptionkey@0"
I20260929 11:38:38.464808 28453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:38:38.468315 28468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:38.469669 28453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:38:38.469843 28453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "fece4c8f243445d09a55951477d50c24"
format_stamp: "Formatted at 2026-09-29 11:38:38 on dist-test-slave-c0l9"
server_key: "637dcef7b0c695aa23c9fd73152e557c"
server_key_iv: "3cdbb225417c7119c42ce2e6ef88afee"
server_key_version: "encryptionkey@0"
I20260929 11:38:38.470055 28453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:38.475524 28453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:38.479377 28453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:38.479635 28453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:38.497172 28453 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:40895
I20260929 11:38:38.497217 28519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:40895 every 8 connection(s)
I20260929 11:38:38.498533 28453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:38.500730 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28453
I20260929 11:38:38.500919 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:38.501340 6183 external_mini_cluster.cc:1428] Setting key 4957e4dd9aecbf8009e3d7593f047f56
I20260929 11:38:38.502333 28520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:38.509438 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:38.502977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52693 (local address 127.6.9.254:40895)
0929 11:38:38.503746 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:38.503761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:38.503811 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:38:38.505616 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:38.505646 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:38.506104 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:38.506542 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:38.507493 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:38.507501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:38.507517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:38.507535 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:38.507618 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:38.507642 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:38.507854 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:38.507981 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 11:38:38.508191 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":312,"threads_started":1}
I20260929 11:38:38.513031 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:38.516634 28520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24: Bootstrap starting.
I20260929 11:38:38.520534 28520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:38.520781 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:38.518114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38599 (local address 127.6.9.254:40895)
0929 11:38:38.518304 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:38.518311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:38.518740 (+ 429us) server_negotiation.cc:407] Connection header received
0929 11:38:38.518889 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:38.518894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:38.518953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:38.519058 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:38.519922 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:38.519928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:38.519932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:38:38.519967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:38.520016 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:38:38.520044 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:38.520164 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:38.520356 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 11:38:38.520422 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:38:38.521889 28520 log.cc:824] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:38.525137 28520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24: No bootstrap required, opened a new log
I20260929 11:38:38.532464 28520 raft_consensus.cc:348] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } }
I20260929 11:38:38.532917 28520 raft_consensus.cc:374] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:38:38.533020 28520 raft_consensus.cc:729] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fece4c8f243445d09a55951477d50c24, State: Initialized, Role: FOLLOWER
I20260929 11:38:38.533691 28520 consensus_queue.cc:260] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } }
I20260929 11:38:38.533990 28520 raft_consensus.cc:388] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:38:38.534129 28520 raft_consensus.cc:482] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:38:38.534289 28520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:38.536458 28520 raft_consensus.cc:504] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } }
I20260929 11:38:38.536972 28520 leader_election.cc:302] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fece4c8f243445d09a55951477d50c24; no voters:
I20260929 11:38:38.537560 28520 leader_election.cc:288] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:38:38.537707 28530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:38.538405 28530 raft_consensus.cc:686] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [term 1 LEADER]: Becoming Leader. State: Replica: fece4c8f243445d09a55951477d50c24, State: Running, Role: LEADER
I20260929 11:38:38.538921 28530 consensus_queue.cc:237] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } }
I20260929 11:38:38.539610 28520 sys_catalog.cc:564] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:38.541639 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fece4c8f243445d09a55951477d50c24. Latest consensus state: current_term: 1 leader_uuid: "fece4c8f243445d09a55951477d50c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } } }
I20260929 11:38:38.542119 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:38.542021 28531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fece4c8f243445d09a55951477d50c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fece4c8f243445d09a55951477d50c24" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 40895 } } }
I20260929 11:38:38.542346 28531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:38.545207 28536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:38.550012 28536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:38.559110 28536 catalog_manager.cc:1347] Generated new cluster ID: eb992f851a3f40c6a6fa0f4ad96c34a3
I20260929 11:38:38.559234 28536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:38.590018 28536 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:38.590449 28536 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:38.596359 28536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fece4c8f243445d09a55951477d50c24: Generated new TSK 0
I20260929 11:38:38.597239 28536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:38.603464 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28453
W20260929 11:38:38.605168 28515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52693
W20260929 11:38:38.605899 6183 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:38:38.606053 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28453
2026-09-29T11:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1777 ms)
[----------] 18 tests from AuthTokenIssuingTest (33164 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:38 dist-test-slave-c0l9 krb5kdc[28551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:38 dist-test-slave-c0l9 krb5kdc[28551](info): set up 2 sockets
Sep 29 11:38:38 dist-test-slave-c0l9 krb5kdc[28551](info): commencing operation
krb5kdc: starting...
W20260929 11:38:39.621045 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:39 dist-test-slave-c0l9 krb5kdc[28551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:39.872567 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42685
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:40.203397 28564 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:40.203763 28564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:40.203864 28564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:40.203969 28564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:40.211450 28564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:40.211561 28564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:40.211630 28564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:40.211710 28564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:40.211796 28564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:40.222131 28564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:40.223532 28564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:40.224969 28564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:40.236931 28569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:40.236883 28570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:40.236850 28572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:40.237789 28564 server_base.cc:1034] running on GCE node
I20260929 11:38:40.238632 28564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:40.240106 28564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:40.241338 28564 hybrid_clock.cc:648] HybridClock initialized: now 1790681920241271 us; error 40 us; skew 500 ppm
I20260929 11:38:40.241669 28564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:40 dist-test-slave-c0l9 krb5kdc[28551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:40.245059 28564 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:40.246641 28564 webserver.cc:460] Webserver started at http://127.6.9.254:41383/ using document root <none> and password file <none>
I20260929 11:38:40.247349 28564 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:40.247455 28564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:40.247771 28564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:40.250875 28564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "7c66460096bc46b4b1dbcb2292d5c1f2"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "7990a8f8eca6b7602479f7740ed723db"
server_key_iv: "6d55e89daed91a49feebff87e55f3c7f"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.251674 28564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "7c66460096bc46b4b1dbcb2292d5c1f2"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "7990a8f8eca6b7602479f7740ed723db"
server_key_iv: "6d55e89daed91a49feebff87e55f3c7f"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.257197 28564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:38:40.260926 28579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:40.262316 28564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:38:40.262490 28564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "7c66460096bc46b4b1dbcb2292d5c1f2"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "7990a8f8eca6b7602479f7740ed723db"
server_key_iv: "6d55e89daed91a49feebff87e55f3c7f"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.262702 28564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:40.274947 28564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:40.278776 28564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:40.279030 28564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:40.296260 28564 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:42685
I20260929 11:38:40.296288 28630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:42685 every 8 connection(s)
I20260929 11:38:40.297616 28564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:40.301456 28631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:40.306674 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28564
I20260929 11:38:40.306875 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:40.307498 6183 external_mini_cluster.cc:1428] Setting key 53ba82d2c68c9d4a0e53dd5e24fd09f1
I20260929 11:38:40.308488 28631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:40 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.253@KRBTEST.COM; defaulting to no policy
I20260929 11:38:40.332724 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.311887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.6.9.254:42685)
0929 11:38:40.313596 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:40.313605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:40.313661 (+ 56us) server_negotiation.cc:407] Connection header received
0929 11:38:40.315268 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:40.315276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:40.315578 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:40.315791 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:40.317346 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.318548 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.320395 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.321032 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.321165 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.325133 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:40.325145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:40.325160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:40.325194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:40.328189 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.329098 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.329105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.329107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.329276 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.329764 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.329768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.329770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.330406 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:40.330573 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:40.331298 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 11:38:40.331581 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1504,"spinlock_wait_cycles":20864,"thread_start_us":1294,"threads_started":1}
I20260929 11:38:40.332784 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.311888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:42685 (local address 127.0.0.1:48036)
0929 11:38:40.312856 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.312922 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:40.314872 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:40.315821 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:40.315827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:40.316597 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:40.317117 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.317145 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.318710 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.318715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.320250 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.320265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.321271 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.321281 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.321440 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.322075 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:40.322104 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:40.324873 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:40.328362 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.328376 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.328390 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.328966 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.329388 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.329393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.329397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.329635 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.330559 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:40.330569 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:40.330855 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 11:38:40.331137 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 11:38:40.331738 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":70,"thread_start_us":371,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.253@KRBTEST.COM" created.
I20260929 11:38:40.335749 28631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:40.336920 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.336484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57112)
0929 11:38:40.336683 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.336803 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 11:38:40.337252 28581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
I20260929 11:38:40.338833 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.338456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57116)
0929 11:38:40.338592 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.338689 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 11:38:40.339118 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:36887: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
I20260929 11:38:40.364027 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:36887
--webserver_interface=127.6.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:38:40.388469 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } attempt: 1
I20260929 11:38:40.389698 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.389238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57124)
0929 11:38:40.389453 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.389548 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 11:38:40.390693 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.390345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57126)
0929 11:38:40.390490 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.390560 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 11:38:40.390893 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:36887: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
I20260929 11:38:40.459192 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } attempt: 2
I20260929 11:38:40.460449 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.459938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57136)
0929 11:38:40.460203 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.460306 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:38:40.461557 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.461186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57152)
0929 11:38:40.461345 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.461430 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 11:38:40.461762 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:36887: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
I20260929 11:38:40.553020 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } attempt: 3
I20260929 11:38:40.554288 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.553830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57160)
0929 11:38:40.554037 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.554135 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:38:40.555335 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.554961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:57164)
0929 11:38:40.555093 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.555187 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:38:40.555531 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:36887: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
W20260929 11:38:40.694357 28638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:40.694726 28638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:40.694799 28638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:40.694883 28638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 11:38:40.698782 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } attempt: 4
I20260929 11:38:40.700114 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.699490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:42606)
0929 11:38:40.699873 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.699974 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260929 11:38:40.701258 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.700843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:42622)
0929 11:38:40.701021 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.701117 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 11:38:40.701483 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:36887: Network error: Client connection negotiation failed: client connection to 127.6.9.253:36887: connect: Connection refused (error 111)
W20260929 11:38:40.702986 28638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:40.703079 28638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:40.703141 28638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:40.703198 28638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:40.703261 28638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:40.713706 28638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:40.715021 28638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:40.716346 28638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:40.728152 28646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:40.728190 28643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:40.728221 28644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:40.728988 28638 server_base.cc:1034] running on GCE node
I20260929 11:38:40.729761 28638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:40.731199 28638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:40.732424 28638 hybrid_clock.cc:648] HybridClock initialized: now 1790681920732350 us; error 47 us; skew 500 ppm
I20260929 11:38:40.732777 28638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:40 dist-test-slave-c0l9 krb5kdc[28551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:40.735786 28638 init.cc:377] Logged in from keytab as kudu/127.6.9.253@KRBTEST.COM (short username kudu)
I20260929 11:38:40.737366 28638 webserver.cc:460] Webserver started at http://127.6.9.253:46173/ using document root <none> and password file <none>
I20260929 11:38:40.738080 28638 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:40.738188 28638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:40.738487 28638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:40.741541 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data/instance:
uuid: "87566e8fdf12434c9946821fbc626af4"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "29b434c4800d0bfc3a6be0200691fc31"
server_key_iv: "25228cb630fc677ccd171eddd47f048d"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.742349 28638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal/instance:
uuid: "87566e8fdf12434c9946821fbc626af4"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "29b434c4800d0bfc3a6be0200691fc31"
server_key_iv: "25228cb630fc677ccd171eddd47f048d"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.747599 28638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:38:40.751070 28653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:40.752457 28638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 11:38:40.752632 28638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "87566e8fdf12434c9946821fbc626af4"
format_stamp: "Formatted at 2026-09-29 11:38:40 on dist-test-slave-c0l9"
server_key: "29b434c4800d0bfc3a6be0200691fc31"
server_key_iv: "25228cb630fc677ccd171eddd47f048d"
server_key_version: "encryptionkey@0"
I20260929 11:38:40.752892 28638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:40.773525 28638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:40.777817 28638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:40.778105 28638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:40.797250 28638 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.253:36887
I20260929 11:38:40.797276 28704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.253:36887 every 8 connection(s)
I20260929 11:38:40.798683 28638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
I20260929 11:38:40.802503 28705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:40.808769 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28638
I20260929 11:38:40.808948 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-1/wal/instance
I20260929 11:38:40.809379 6183 external_mini_cluster.cc:1428] Setting key 039e1eeeaa2721d61041ca0a2cbbd61b
I20260929 11:38:40.809568 28705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:40 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:40.830539 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.812213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.6.9.254:42685)
0929 11:38:40.812444 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:40.812451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:40.813746 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 11:38:40.815688 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:40.815694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:40.815779 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:40.815901 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:40.817489 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.818492 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.819957 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.820550 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.820703 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.824275 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:40.824281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:40.824285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:40.824321 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:40.826911 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.827918 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.827922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.827924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.828150 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.828655 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.828658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.828689 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.829036 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:40.829157 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:40.830109 (+ 952us) server_negotiation.cc:299] Negotiation successful
0929 11:38:40.830306 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.6.9.252@KRBTEST.COM; defaulting to no policy
I20260929 11:38:40.832118 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.812731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:42685 (local address 127.0.0.1:59876)
0929 11:38:40.813548 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.813597 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:40.815406 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:40.816092 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:40.816100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:40.816655 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:40.817278 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.817301 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.818637 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.818641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.819823 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.819835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.820774 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.820780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.820998 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.821655 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:40.821677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:40.824022 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:40.827083 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.827093 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.827114 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.827765 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.828276 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.828281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.828284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.828548 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.829172 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:40.829182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:40.829692 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 11:38:40.830035 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 11:38:40.830612 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":320,"threads_started":1}
I20260929 11:38:40.834015 28705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.252@KRBTEST.COM" created.
Sep 29 11:38:40 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:40.849367 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.834834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:42628)
0929 11:38:40.835042 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.835063 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:40.835194 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:40.836028 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:40.836032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:40.836424 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:40.836569 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.836577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.838366 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.838371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.839444 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:40.839457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.840474 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.840478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:40.840621 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.841251 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:40.841269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:40.843673 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:40.846750 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.846755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.846759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.847125 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.847490 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:40.847495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:40.847499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:40.847629 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:40.848340 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:40.848345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:40.848528 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 11:38:40.848647 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:38:40.848955 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:38:40.849416 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.834896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.6.9.253:36887)
0929 11:38:40.835500 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:40.835509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:40.835539 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:38:40.835591 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:40.835598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:40.835834 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:40.835998 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:40.836763 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.838209 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.839581 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:40.840337 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:40.840450 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:40.843881 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:40.843893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:40.843908 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:40.843939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:40.846626 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.847244 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.847250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.847253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.847392 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:40.847722 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:40.847726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:40.847728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:40.848197 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:40.848439 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:40.848832 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 11:38:40.849046 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":281,"threads_started":1}
I20260929 11:38:40.852046 28705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:40.853274 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.852806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44950)
0929 11:38:40.852986 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.853094 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:40.853570 28657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:40.855120 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.854815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44964)
0929 11:38:40.854945 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.855021 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:40.855340 28705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
I20260929 11:38:40.866050 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:45657
--webserver_interface=127.6.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:38:40.904693 28705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 1
I20260929 11:38:40.905933 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.905463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44966)
0929 11:38:40.905666 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.905770 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 11:38:40.906924 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.906580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44978)
0929 11:38:40.906733 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.906818 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:40.907119 28705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:40.975333 28705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 2
I20260929 11:38:40.976476 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.976043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44988)
0929 11:38:40.976229 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.976321 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:38:40.977456 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:40.977134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:44994)
0929 11:38:40.977274 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:40.977342 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:38:40.977650 28705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:41.000784 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } attempt: 5
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:41.014282 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.001519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:42638)
0929 11:38:41.001715 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.001736 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.001879 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.002238 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.002244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.002648 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.002831 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.002842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.003877 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.003881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.004980 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.004996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.005758 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.005764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.005963 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.006549 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.006568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.008778 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.011813 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.011819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.011823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.012206 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.012581 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.012586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.012590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.012794 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.013318 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.013323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.013483 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.013600 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.013916 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 11:38:41.014287 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.001690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.6.9.253:36887)
0929 11:38:41.001895 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.001901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.001924 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:38:41.002001 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.002006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.002086 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.002168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.003000 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.003718 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.005151 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.005605 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.005722 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.008994 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.008998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.009002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.009039 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.011641 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.012330 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.012335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.012337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.012467 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.012897 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.012901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.012903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.013197 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.013344 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.013730 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.013952 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:38:41.015692 28631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:41.016932 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.016477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45006)
0929 11:38:41.016647 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.016779 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:38:41.017765 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.017461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45016)
0929 11:38:41.017591 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.017657 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:38:41.017954 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:41.068857 28705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 3
I20260929 11:38:41.069204 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 1
I20260929 11:38:41.070092 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.069523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45030)
0929 11:38:41.069777 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.069893 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 11:38:41.070170 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.069768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45040)
0929 11:38:41.069935 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.070012 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 11:38:41.071333 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.070783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45044)
0929 11:38:41.071052 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.071167 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 11:38:41.071439 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.070752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45046)
0929 11:38:41.071159 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.071269 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"threads_started":1}
W20260929 11:38:41.071662 28705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
W20260929 11:38:41.071985 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:41.140235 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 2
I20260929 11:38:41.141656 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45060)
0929 11:38:41.141203 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.141392 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 11:38:41.142700 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.142326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45066)
0929 11:38:41.142470 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.142553 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 11:38:41.142930 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
W20260929 11:38:41.193488 28712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:41.193795 28712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:41.193866 28712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:41.193946 28712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:41.201023 28712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:41.201115 28712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:41.201193 28712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:41.201246 28712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:41.201306 28712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:41.210953 28712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:42685,127.6.9.253:36887,127.6.9.252:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:41.212210 28712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:41.213496 28712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:38:41.214866 28705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 4
I20260929 11:38:41.216037 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.215552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45080)
0929 11:38:41.215790 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.215881 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:38:41.217043 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.216749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45096)
0929 11:38:41.216866 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.216938 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 11:38:41.217244 28705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
W20260929 11:38:41.225728 28721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:41.225723 28719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:41.225682 28718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:41.226490 28712 server_base.cc:1034] running on GCE node
I20260929 11:38:41.227236 28712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:41.228646 28712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:41.229905 28712 hybrid_clock.cc:648] HybridClock initialized: now 1790681921229829 us; error 50 us; skew 500 ppm
I20260929 11:38:41.230262 28712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681921, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:41.233544 28712 init.cc:377] Logged in from keytab as kudu/127.6.9.252@KRBTEST.COM (short username kudu)
I20260929 11:38:41.235196 28712 webserver.cc:460] Webserver started at http://127.6.9.252:43617/ using document root <none> and password file <none>
I20260929 11:38:41.235985 28712 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:41.236102 28712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:41.236451 28712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:41.239805 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data/instance:
uuid: "229e27e8400d49d69c79a538ed893809"
format_stamp: "Formatted at 2026-09-29 11:38:41 on dist-test-slave-c0l9"
server_key: "fabba98a80370a52b5c7d88d39b45d22"
server_key_iv: "7a2d11666ef33979bad7362e693b3367"
server_key_version: "encryptionkey@0"
I20260929 11:38:41.240711 28712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal/instance:
uuid: "229e27e8400d49d69c79a538ed893809"
format_stamp: "Formatted at 2026-09-29 11:38:41 on dist-test-slave-c0l9"
server_key: "fabba98a80370a52b5c7d88d39b45d22"
server_key_iv: "7a2d11666ef33979bad7362e693b3367"
server_key_version: "encryptionkey@0"
I20260929 11:38:41.246132 28712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:38:41.249140 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 3
I20260929 11:38:41.249922 28728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:41.250301 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.249853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45100)
0929 11:38:41.250041 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.250137 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:38:41.251370 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.250993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45114)
0929 11:38:41.251168 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.251238 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 11:38:41.251578 28631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:45657: Network error: Client connection negotiation failed: client connection to 127.6.9.252:45657: connect: Connection refused (error 111)
I20260929 11:38:41.251641 28712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:38:41.251811 28712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "229e27e8400d49d69c79a538ed893809"
format_stamp: "Formatted at 2026-09-29 11:38:41 on dist-test-slave-c0l9"
server_key: "fabba98a80370a52b5c7d88d39b45d22"
server_key_iv: "7a2d11666ef33979bad7362e693b3367"
server_key_version: "encryptionkey@0"
I20260929 11:38:41.252012 28712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:41.263422 28712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:41.267562 28712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:41.267830 28712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:41.284999 28712 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.252:45657
I20260929 11:38:41.285018 28779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.252:45657 every 8 connection(s)
I20260929 11:38:41.286612 28712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
I20260929 11:38:41.289580 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28712
I20260929 11:38:41.289770 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681566518919-6183-0/minicluster-data/master-2/wal/instance
I20260929 11:38:41.290294 6183 external_mini_cluster.cc:1428] Setting key d09183a0aa1d20789fedf2a7139e7708
I20260929 11:38:41.290524 28780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:41.300137 28780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681921, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:41.312410 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.292082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.6.9.252:45657)
0929 11:38:41.292817 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.292828 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.292867 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:38:41.294893 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.294913 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.295301 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.295642 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.297172 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.298416 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.299636 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.300327 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.300497 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.304957 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.304969 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.304987 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.305033 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.308096 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.308888 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.308895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.308903 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.309089 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.309446 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.309450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.309453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.310119 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.310390 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.310824 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.311170 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":521,"thread_start_us":330,"threads_started":1}
I20260929 11:38:41.320271 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.303275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:42685 (local address 127.0.0.1:59892)
0929 11:38:41.304810 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.304853 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.305018 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.305360 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.305367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.306142 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.306440 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.306453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.307430 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.307436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.308740 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.308755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.309629 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.309634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.309762 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.310398 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.310421 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.313931 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.317734 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.317744 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.317750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.318274 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.318662 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.318667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.318670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.318826 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.319397 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.319403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.319561 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.319722 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.319953 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":2,"thread_start_us":442,"threads_started":1}
I20260929 11:38:41.320493 28634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.303296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.6.9.254:42685)
0929 11:38:41.303526 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.303533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.304943 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 11:38:41.305144 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.305148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.305211 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.305299 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.306591 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.307284 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.308879 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.309458 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.309583 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.314172 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.314177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.314181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.314218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.317599 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.318394 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.318398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.318400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.318555 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.318923 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.318926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.318928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.319266 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.319435 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.319878 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.320108 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:38:41.321813 28780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681921, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:41.331871 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.315547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.6.9.254:42685)
0929 11:38:41.316157 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.316162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.316179 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:38:41.316220 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.316223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.316278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.316367 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.317897 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.319020 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.320949 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.321469 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.321560 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.325919 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.325924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.325927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.325959 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.328933 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.329645 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.329649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.329652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.329778 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.330232 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.330238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.330241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.330645 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.330930 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.331229 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.331482 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260929 11:38:41.335464 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.322528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:36887 (local address 127.0.0.1:42648)
0929 11:38:41.322708 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.322727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.322820 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.323104 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.323109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.323525 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.323726 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.323738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.324761 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.324765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.325671 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.325684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.326374 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.326379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.326561 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.326996 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.327011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.329218 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.332834 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.332839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.332843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.333286 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.333761 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.333766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.333769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.333952 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.334518 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.334522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.334681 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.334800 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.335093 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:38:41.335486 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.322551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.6.9.253:36887)
0929 11:38:41.322743 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.322749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.322767 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:38:41.322922 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.322926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.322984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.323056 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.323875 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.324585 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.325814 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.326231 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.326341 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.329460 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.329466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.329470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.329511 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.332635 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.333427 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.333432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.333435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.333628 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.334067 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.334073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.334076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.334379 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.334541 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.334917 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.335135 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:38:41.336479 28780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681921, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:38:41.348068 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.333595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.6.9.253:36887)
0929 11:38:41.334139 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.334143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.334161 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:38:41.334200 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.334204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.334254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.334345 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.335553 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.336539 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.337936 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.338391 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.338495 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.342938 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.342943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.342946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.342973 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.345670 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.346253 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.346259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.346263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.346394 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.346731 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.346736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.346739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.346989 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.347222 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.347506 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.347718 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":236,"threads_started":1}
I20260929 11:38:41.349957 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.337287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45134)
0929 11:38:41.337550 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.337567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.337673 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.338681 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.338685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.339127 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.339303 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.339316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.340246 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.340251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.341568 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.341581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.342321 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.342325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.342483 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.342951 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.342971 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.345071 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.347672 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.347677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.347681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.348099 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.348398 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.348403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.348406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.348535 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.349111 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.349115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.349265 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.349385 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.349646 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 11:38:41.350224 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.337314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.6.9.252:45657)
0929 11:38:41.337458 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.337464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.337631 (+ 167us) server_negotiation.cc:407] Connection header received
0929 11:38:41.337769 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.337772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.337828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.337904 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.339455 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.340110 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.341685 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.342164 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.342326 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.345237 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.345242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.345245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.345273 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.347552 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.348194 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.348199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.348202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.348306 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.348648 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.348652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.348654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.348996 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.349129 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.349503 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.349843 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":30}
I20260929 11:38:41.353080 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:41.356125 28780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: Bootstrap starting.
I20260929 11:38:41.360195 28780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:41.356513 28606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894
I20260929 11:38:41.361492 28780 log.cc:824] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:41.364399 28780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: No bootstrap required, opened a new log
I20260929 11:38:41.374781 28780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.375166 28780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e27e8400d49d69c79a538ed893809, State: Initialized, Role: FOLLOWER
I20260929 11:38:41.375651 28780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.376593 28788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.376982 28788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.377656 28780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:38:41.386731 28799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:38:41.386857 28799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:38:41.384498 28712 master_runner.cc:186] Error getting master registration for 127.6.9.254:42685: OK, instance_id { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" instance_seqno: 1790681920291250 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:38:41.395203 28712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:38:41.428761 28631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 4
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:38:41.449759 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.429609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.6.9.252:45657)
0929 11:38:41.430076 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.430080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.430095 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:38:41.430144 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.430147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.430217 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.430304 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.431455 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.432604 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.434504 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.435208 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.435350 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.439370 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.439377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.439382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.439423 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.446842 (+ 7419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.447706 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.447710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.447713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.447865 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.448263 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.448267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.448270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.448728 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.448868 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.449310 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.449484 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260929 11:38:41.449937 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.429525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45138)
0929 11:38:41.429755 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.429774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.429893 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.430396 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.430402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.431007 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.431297 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.431312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.432809 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.432815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.434342 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.434359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.435412 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.435419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.435609 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.436299 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.436324 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.439103 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.447007 (+ 7904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.447014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.447019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.447571 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.447964 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.447969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.447973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.448162 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.448852 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.448858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.449038 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.449198 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.449512 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 11:38:41.456063 28631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: Bootstrap starting.
I20260929 11:38:41.460295 28631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:41.461758 28631 log.cc:824] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:41.465355 28631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: No bootstrap required, opened a new log
I20260929 11:38:41.473272 28631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.473760 28631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c66460096bc46b4b1dbcb2292d5c1f2, State: Initialized, Role: FOLLOWER
I20260929 11:38:41.474298 28631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.475337 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.475703 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.476730 28631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:41.476799 28606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:38:41.476989 28606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:41.479746 28606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.481943 28606 leader_election.cc:288] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [CANDIDATE]: Term 1 election: Requested vote from peers 87566e8fdf12434c9946821fbc626af4 (127.6.9.253:36887), 229e27e8400d49d69c79a538ed893809 (127.6.9.252:45657)
I20260929 11:38:41.481982 28755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "229e27e8400d49d69c79a538ed893809"
I20260929 11:38:41.482468 28755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:41.486057 28755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c66460096bc46b4b1dbcb2292d5c1f2 in term 1.
I20260929 11:38:41.486717 28583 leader_election.cc:302] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229e27e8400d49d69c79a538ed893809, 7c66460096bc46b4b1dbcb2292d5c1f2; no voters:
I20260929 11:38:41.487206 28802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:41.492936 28802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [term 1 LEADER]: Becoming Leader. State: Replica: 7c66460096bc46b4b1dbcb2292d5c1f2, State: Running, Role: LEADER
I20260929 11:38:41.493574 28802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.501955 28810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c66460096bc46b4b1dbcb2292d5c1f2. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.502434 28810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:41.512730 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.485853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.6.9.254:42685)
0929 11:38:41.486549 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.486555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.489004 (+ 2449us) server_negotiation.cc:407] Connection header received
0929 11:38:41.489308 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.489315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.489424 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.489499 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.491963 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.493155 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.496281 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.496933 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.497027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.503262 (+ 6235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.503268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.503284 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.503328 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.507021 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.510290 (+ 3269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.510296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.510299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.510481 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.510913 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.510918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.510921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.511289 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.511531 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.512086 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.512346 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531}
I20260929 11:38:41.515923 28815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:41.516433 28705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } attempt: 5
W20260929 11:38:41.520012 28734 tablet.cc:2369] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:38:41.523882 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.486855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.6.9.253:36887)
0929 11:38:41.487072 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.487078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.496753 (+ 9675us) server_negotiation.cc:407] Connection header received
0929 11:38:41.497125 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.497131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.497216 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.497287 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.500799 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.501930 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.504924 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.505561 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.505659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.510855 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.510862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.510867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.510910 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.514682 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.518992 (+ 4310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.519000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.519004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.519191 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.521443 (+ 2252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.521451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.521456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.521921 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.522207 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.523170 (+ 963us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.523471 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 11:38:41.531908 28585 tablet.cc:2369] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:38:41 dist-test-slave-c0l9 krb5kdc[28551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681920, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:38:41.544824 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.487666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.6.9.252:45657)
0929 11:38:41.487831 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.487837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.502784 (+ 14947us) server_negotiation.cc:407] Connection header received
0929 11:38:41.507209 (+ 4425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.507216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.507478 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.507572 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.513476 (+ 5904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.514537 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.516336 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.516998 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.517096 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.532051 (+ 14955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.532061 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.532067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.532121 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.536107 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.537273 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.537279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.537282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.537458 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.537961 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.537965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.537968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.538324 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.538587 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.544083 (+ 5496us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.544412 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:38:41.544936 28815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:41.555751 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.517421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.6.9.252:45657)
0929 11:38:41.533414 (+ 15993us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.533422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.533476 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:38:41.533578 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.533583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.533762 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.535101 (+ 1339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.535630 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.537869 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.539845 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.540976 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.541872 (+ 896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.546060 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:41.546066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:41.546071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:41.546109 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:41.549906 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.551191 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.551197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.551200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.551475 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:41.552206 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:41.552212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:41.552220 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:41.552932 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:41.553175 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:41.554010 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 11:38:41.554403 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15712,"thread_start_us":18782,"threads_started":1}
I20260929 11:38:41.557159 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:41.517000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:45657 (local address 127.0.0.1:45158)
0929 11:38:41.517262 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:41.517281 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:41.517376 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:41.534087 (+ 16711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:41.534093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:41.534604 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:41.534872 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.534888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.538099 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.538105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.539467 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:41.539487 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:41.541231 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:41.541239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:41.541455 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:41.542170 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:41.542198 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:41.545474 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:41.550157 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.550162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.550165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.551002 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.551723 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:41.551728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:41.551731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:41.552020 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:41.553315 (+ 1295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:41.553320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:41.553579 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 11:38:41.554656 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0929 11:38:41.555704 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 11:38:41.563278 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: Bootstrap starting.
I20260929 11:38:41.566736 28755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [term 1 FOLLOWER]: Refusing update from remote peer 7c66460096bc46b4b1dbcb2292d5c1f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:41.567991 28810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:41.569360 28705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:41.571887 28705 log.cc:824] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:41.585531 28705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: No bootstrap required, opened a new log
I20260929 11:38:41.576265 28564 master_runner.cc:186] Error getting master registration for 127.6.9.253:36887: OK, instance_id { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" instance_seqno: 1790681920791791 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:38:41.590631 28564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:38:41.598476 28705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.598928 28705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87566e8fdf12434c9946821fbc626af4, State: Initialized, Role: FOLLOWER
I20260929 11:38:41.599443 28705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } }
I20260929 11:38:41.600203 28788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c66460096bc46b4b1dbcb2292d5c1f2. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.600495 28788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.601934 28824 mvcc.cc:205] Tried to move back new op lower bound from 7334633150719610880 to 7334633150443286528. Current Snapshot: MvccSnapshot[applied={T|T < 7334633150719610880}]
I20260929 11:38:41.604061 28788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.604292 28788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.604487 28828 mvcc.cc:205] Tried to move back new op lower bound from 7334633150719610880 to 7334633150443286528. Current Snapshot: MvccSnapshot[applied={T|T < 7334633150719610880}]
I20260929 11:38:41.609500 28830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.609941 28830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.610530 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.610849 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:41.611829 28705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:41.611915 28680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "87566e8fdf12434c9946821fbc626af4"
I20260929 11:38:41.612421 28680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:41.613677 28815 catalog_manager.cc:1347] Generated new cluster ID: 0a71c8950325403cacb1fc4f5c0cb9cf
I20260929 11:38:41.613777 28815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:41.615882 28680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c66460096bc46b4b1dbcb2292d5c1f2 in term 1.
I20260929 11:38:41.618100 28679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [term 1 FOLLOWER]: Refusing update from remote peer 7c66460096bc46b4b1dbcb2292d5c1f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:41.619076 28810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:41.656883 28815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:41.658999 28815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:38:41.688416 28830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c66460096bc46b4b1dbcb2292d5c1f2. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.688791 28830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.689881 28802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.690178 28802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:41.691030 28840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c66460096bc46b4b1dbcb2292d5c1f2" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 42685 } } peers { permanent_uuid: "87566e8fdf12434c9946821fbc626af4" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 36887 } } peers { permanent_uuid: "229e27e8400d49d69c79a538ed893809" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 45657 } } }
I20260929 11:38:41.691505 28840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:41.692951 28815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c66460096bc46b4b1dbcb2292d5c1f2: Generated new TSK 0
I20260929 11:38:41.694001 28815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:38:41.696282 28847 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 11:38:41.762943 28785 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.760283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.6.9.254:42685)
0929 11:38:41.760478 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.760483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.760831 (+ 348us) server_negotiation.cc:407] Connection header received
0929 11:38:41.761182 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.761188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.761279 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.761361 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.762631 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59916: BlockingRecv error: recv got EOF from 127.0.0.1:59916 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 11:38:41.765228 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.761063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.6.9.253:36887)
0929 11:38:41.761235 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.761241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.763467 (+ 2226us) server_negotiation.cc:407] Connection header received
0929 11:38:41.763713 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.763719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.763810 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.763881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.764946 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42668: BlockingRecv error: recv got EOF from 127.0.0.1:42668 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 11:38:41.769945 28817 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.767777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.6.9.252:45657)
0929 11:38:41.767984 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.767990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.768140 (+ 150us) server_negotiation.cc:407] Connection header received
0929 11:38:41.768403 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.768409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.768518 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.768593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.769674 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45166: BlockingRecv error: recv got EOF from 127.0.0.1:45166 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 11:38:41.778420 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.776597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.6.9.253:36887)
0929 11:38:41.776800 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.776806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.776867 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:38:41.777195 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.777202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.777315 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.777407 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.778186 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42682: BlockingRecv error: recv got EOF from 127.0.0.1:42682 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 11:38:41.790683 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.784200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.6.9.253:36887)
0929 11:38:41.784386 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.784392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.789286 (+ 4894us) server_negotiation.cc:407] Connection header received
0929 11:38:41.789496 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.789502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.789580 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.789666 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.790434 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42690: BlockingRecv error: recv got EOF from 127.0.0.1:42690 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 11:38:41.797406 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.795332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.6.9.253:36887)
0929 11:38:41.795513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.795519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.796004 (+ 485us) server_negotiation.cc:407] Connection header received
0929 11:38:41.796192 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.796198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.796273 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.796356 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.797145 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42698: BlockingRecv error: recv got EOF from 127.0.0.1:42698 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 11:38:41.852798 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.850441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.6.9.253:36887)
0929 11:38:41.850648 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.850654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.850969 (+ 315us) server_negotiation.cc:407] Connection header received
0929 11:38:41.851232 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.851238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.851341 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.851425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.852505 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42700: BlockingRecv error: recv got EOF from 127.0.0.1:42700 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 11:38:41.869417 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.866413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.6.9.253:36887)
0929 11:38:41.866651 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.866658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.867287 (+ 629us) server_negotiation.cc:407] Connection header received
0929 11:38:41.867748 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.867772 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.867872 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.867958 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.869191 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42704: BlockingRecv error: recv got EOF from 127.0.0.1:42704 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260929 11:38:41.893543 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.891099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.6.9.253:36887)
0929 11:38:41.891324 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.891331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.891925 (+ 594us) server_negotiation.cc:407] Connection header received
0929 11:38:41.892131 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.892137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.892237 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.892357 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.893294 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42712: BlockingRecv error: recv got EOF from 127.0.0.1:42712 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 11:38:41.932343 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:41.929720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.6.9.253:36887)
0929 11:38:41.929938 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:41.929945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:41.930515 (+ 570us) server_negotiation.cc:407] Connection header received
0929 11:38:41.930824 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:41.930831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:41.930929 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:41.931016 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:41.932084 (+ 1068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42720: BlockingRecv error: recv got EOF from 127.0.0.1:42720 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 11:38:42.016422 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:42.014443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.6.9.253:36887)
0929 11:38:42.014621 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.014627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.015030 (+ 403us) server_negotiation.cc:407] Connection header received
0929 11:38:42.015249 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.015255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.015333 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.015417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:42.016189 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42726: BlockingRecv error: recv got EOF from 127.0.0.1:42726 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:38:42.150868 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:42.148789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.6.9.253:36887)
0929 11:38:42.149056 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.149062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.149391 (+ 329us) server_negotiation.cc:407] Connection header received
0929 11:38:42.149622 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.149628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.149726 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.149831 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:42.150677 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42736: BlockingRecv error: recv got EOF from 127.0.0.1:42736 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 11:38:42.388840 28799 catalog_manager.cc:1259] Loaded cluster ID: 0a71c8950325403cacb1fc4f5c0cb9cf
I20260929 11:38:42.388945 28799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: loading cluster ID for follower catalog manager: success
I20260929 11:38:42.391482 28799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: acquiring CA information for follower catalog manager: success
I20260929 11:38:42.393299 28799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 229e27e8400d49d69c79a538ed893809: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:38:42.413754 28786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:42.411125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.6.9.253:36887)
0929 11:38:42.411347 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.411354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.411924 (+ 570us) server_negotiation.cc:407] Connection header received
0929 11:38:42.412147 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.412153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.412274 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.412386 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:42.413493 (+ 1107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42738: BlockingRecv error: recv got EOF from 127.0.0.1:42738 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:38:42.700914 28847 catalog_manager.cc:1259] Loaded cluster ID: 0a71c8950325403cacb1fc4f5c0cb9cf
I20260929 11:38:42.701035 28847 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: loading cluster ID for follower catalog manager: success
I20260929 11:38:42.703603 28847 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: acquiring CA information for follower catalog manager: success
I20260929 11:38:42.705487 28847 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87566e8fdf12434c9946821fbc626af4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:38:42.939343 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:42.932069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.6.9.253:36887)
0929 11:38:42.932876 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.932881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.932908 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:38:42.932975 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.932978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.933133 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.933243 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:42.933877 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.934720 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.936787 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.937240 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.937338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:42.938044 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:42.938625 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:42.938793 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:42.938885 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 11:38:42.938975 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":570,"threads_started":1}
I20260929 11:38:42.952055 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:42.946207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.9.252:45657)
0929 11:38:42.946842 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.946848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.946866 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:38:42.946967 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.946971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.947128 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.947278 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:42.947726 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.948453 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.949806 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.950223 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.950337 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:42.950835 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:42.951367 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:42.951505 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:42.951689 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 11:38:42.951774 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":381,"threads_started":1}
I20260929 11:38:42.979207 28923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:42.968968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.6.9.254:42685)
0929 11:38:42.969461 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.969467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.970378 (+ 911us) server_negotiation.cc:407] Connection header received
0929 11:38:42.970520 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.970527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.970794 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.970993 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:42.971874 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.973117 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.975400 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.976072 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.976216 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:42.977056 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:42.977643 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:42.977756 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:42.978749 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 11:38:42.978849 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260929 11:38:42.986204 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28564
I20260929 11:38:42.987352 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:42.968955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.6.9.253:36887)
0929 11:38:42.969234 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.969240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.978591 (+ 9351us) server_negotiation.cc:407] Connection header received
0929 11:38:42.979858 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.979866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.979966 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.980046 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:42.981360 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.982496 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.984707 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.985443 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.985557 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:42.986278 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:42.986426 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:42.986494 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:42.986856 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:38:42.986956 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260929 11:38:42.987361 28626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:42.995569 28918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:42.968905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.6.9.252:45657)
0929 11:38:42.969094 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:42.969098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:42.988282 (+ 19184us) server_negotiation.cc:407] Connection header received
0929 11:38:42.988576 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:42.988583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:42.988734 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:42.988808 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:42.989414 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.990494 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.993250 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:42.993923 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:42.994030 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:42.994678 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:42.994844 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:42.994926 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:42.995144 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 11:38:42.995252 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:38:43.107517 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28564
W20260929 11:38:43.124831 28731 connection.cc:535] server connection from 127.0.0.1:45138 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:38:43.125185 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28638
I20260929 11:38:43.126308 28700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:43.232064 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28638
I20260929 11:38:43.248301 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28712
I20260929 11:38:43.249446 28775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:43.357007 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28712
2026-09-29T11:38:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4760 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:38:43 dist-test-slave-c0l9 krb5kdc[28932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:38:43 dist-test-slave-c0l9 krb5kdc[28932](info): set up 2 sockets
Sep 29 11:38:43 dist-test-slave-c0l9 krb5kdc[28932](info): commencing operation
krb5kdc: starting...
W20260929 11:38:44.380947 6183 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/test-user.keytab.
Sep 29 11:38:44 dist-test-slave-c0l9 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:38:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:38:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.9.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
Entry for principal kudu/127.6.9.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab.
I20260929 11:38:44.631757 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33315
--webserver_interface=127.6.9.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:38:44.952910 28945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:44.953291 28945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:44.953375 28945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:44.953457 28945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:44.960683 28945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:44.960791 28945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:44.960867 28945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:44.960919 28945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:44.960978 28945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:44.970741 28945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.254.keytab
--principal=kudu/127.6.9.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.254:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.9.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:44.971987 28945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:44.973266 28945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:44.984529 28953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:44.984562 28951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:44.985061 28945 server_base.cc:1034] running on GCE node
W20260929 11:38:44.984593 28950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:44.985918 28945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:44.987283 28945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:44.988521 28945 hybrid_clock.cc:648] HybridClock initialized: now 1790681924988455 us; error 36 us; skew 500 ppm
I20260929 11:38:44.988881 28945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:44 dist-test-slave-c0l9 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:44.991904 28945 init.cc:377] Logged in from keytab as kudu/127.6.9.254@KRBTEST.COM (short username kudu)
I20260929 11:38:44.993347 28945 webserver.cc:460] Webserver started at http://127.6.9.254:39419/ using document root <none> and password file <none>
I20260929 11:38:44.994007 28945 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:44.994091 28945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:44.994371 28945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:44.997395 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data/instance:
uuid: "0a48099800b548e18ce6c26573da0e58"
format_stamp: "Formatted at 2026-09-29 11:38:44 on dist-test-slave-c0l9"
server_key: "d06a82df0886a395be6b8c6984c6c98f"
server_key_iv: "350cca178025e33db279532a6007251d"
server_key_version: "encryptionkey@0"
I20260929 11:38:44.998143 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance:
uuid: "0a48099800b548e18ce6c26573da0e58"
format_stamp: "Formatted at 2026-09-29 11:38:44 on dist-test-slave-c0l9"
server_key: "d06a82df0886a395be6b8c6984c6c98f"
server_key_iv: "350cca178025e33db279532a6007251d"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.003337 28945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:38:45.006758 28960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:45.008068 28945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 11:38:45.008247 28945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal
uuid: "0a48099800b548e18ce6c26573da0e58"
format_stamp: "Formatted at 2026-09-29 11:38:44 on dist-test-slave-c0l9"
server_key: "d06a82df0886a395be6b8c6984c6c98f"
server_key_iv: "350cca178025e33db279532a6007251d"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.008438 28945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:45.020996 28945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:45.025198 28945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:45.025462 28945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:45.042595 28945 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.254:33315
I20260929 11:38:45.042614 29011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.254:33315 every 8 connection(s)
I20260929 11:38:45.043913 28945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/data/info.pb
I20260929 11:38:45.046208 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 28945
I20260929 11:38:45.046366 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-0/wal/instance
I20260929 11:38:45.046797 6183 external_mini_cluster.cc:1428] Setting key fa40a8f522ac89bf9441a643aeece3a5
I20260929 11:38:45.047554 29012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:45.053831 29012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.9.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.253@KRBTEST.COM" created.
I20260929 11:38:45.076609 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.057145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33315 (local address 127.0.0.1:55292)
0929 11:38:45.058319 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.058358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:45.060091 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:45.061016 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:45.061021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:45.061559 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:45.061905 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.061922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.064047 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.064051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.065178 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.065190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.066389 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.066394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.066512 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.067136 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:45.067162 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:45.069276 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:45.072959 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.072968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.072977 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.073349 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.073803 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.073807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.073809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.073956 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.074768 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:45.074773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:45.074973 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 11:38:45.075167 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 11:38:45.075548 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"spinlock_wait_cycles":15104,"thread_start_us":720,"threads_started":1}
I20260929 11:38:45.076655 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.057127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.6.9.254:33315)
0929 11:38:45.057923 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:45.057935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:45.058487 (+ 552us) server_negotiation.cc:407] Connection header received
0929 11:38:45.060463 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:45.060470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:45.060779 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:45.061001 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:45.062111 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.063874 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.065314 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.066170 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.066359 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.069512 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:45.069521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:45.069533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:45.069566 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:45.072838 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.073489 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.073499 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.073502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.073697 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.074047 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.074051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.074053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.074645 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:45.074824 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:45.075339 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 11:38:45.075717 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":574,"thread_start_us":367,"threads_started":1}
I20260929 11:38:45.079828 29012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:45.080991 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.080583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47368)
0929 11:38:45.080770 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.080885 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:38:45.081241 28962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
I20260929 11:38:45.082600 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.082280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47378)
0929 11:38:45.082415 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.082477 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:45.082791 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43791: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
Entry for principal kudu/127.6.9.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab.
I20260929 11:38:45.101500 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:43791
--webserver_interface=127.6.9.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:38:45.132100 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } attempt: 1
I20260929 11:38:45.133211 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.132783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47386)
0929 11:38:45.132968 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.133063 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:38:45.134217 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.133876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47400)
0929 11:38:45.134021 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.134086 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:38:45.134383 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43791: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
I20260929 11:38:45.202639 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } attempt: 2
I20260929 11:38:45.203804 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.203312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47410)
0929 11:38:45.203554 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.203662 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:38:45.204931 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.204524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47426)
0929 11:38:45.204714 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.204801 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:45.205096 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43791: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
I20260929 11:38:45.296360 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } attempt: 3
I20260929 11:38:45.297580 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.297146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47432)
0929 11:38:45.297343 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.297437 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 11:38:45.298723 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.298299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47448)
0929 11:38:45.298478 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.298566 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 11:38:45.298907 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43791: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
W20260929 11:38:45.424508 29019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:45.424891 29019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:45.424997 29019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:45.425113 29019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:45.432341 29019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:45.432447 29019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:45.432531 29019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:45.432596 29019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:45.432694 29019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:45.442155 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } attempt: 4
I20260929 11:38:45.443255 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.442729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47458)
0929 11:38:45.443013 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.443109 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 11:38:45.444267 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.443929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47470)
0929 11:38:45.444078 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.444149 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:38:45.444439 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.253:43791: Network error: Client connection negotiation failed: client connection to 127.6.9.253:43791: connect: Connection refused (error 111)
I20260929 11:38:45.445685 29019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.253.keytab
--principal=kudu/127.6.9.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.253:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.9.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:45.446966 29019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:45.448236 29019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:45.459100 29024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:45.459182 29025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:45.459205 29027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:45.460352 29019 server_base.cc:1034] running on GCE node
I20260929 11:38:45.461264 29019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:45.462741 29019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:45.464007 29019 hybrid_clock.cc:648] HybridClock initialized: now 1790681925463926 us; error 40 us; skew 500 ppm
I20260929 11:38:45.464430 29019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:45.467617 29019 init.cc:377] Logged in from keytab as kudu/127.6.9.253@KRBTEST.COM (short username kudu)
I20260929 11:38:45.469094 29019 webserver.cc:460] Webserver started at http://127.6.9.253:43915/ using document root <none> and password file <none>
I20260929 11:38:45.469772 29019 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:45.469872 29019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:45.470170 29019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:45.473181 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data/instance:
uuid: "a35d3d1a82f4431c99613e390ee3b19d"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "201b7aa9405b5171844046c3f5e50a14"
server_key_iv: "c5cdf3fa815c9cfba203c896863e586d"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.473985 29019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal/instance:
uuid: "a35d3d1a82f4431c99613e390ee3b19d"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "201b7aa9405b5171844046c3f5e50a14"
server_key_iv: "c5cdf3fa815c9cfba203c896863e586d"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.479436 29019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 11:38:45.483255 29034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:45.484802 29019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:38:45.484988 29019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal
uuid: "a35d3d1a82f4431c99613e390ee3b19d"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "201b7aa9405b5171844046c3f5e50a14"
server_key_iv: "c5cdf3fa815c9cfba203c896863e586d"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.485213 29019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:45.523895 29019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:45.527971 29019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:45.528223 29019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:45.546018 29019 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.253:43791
I20260929 11:38:45.546042 29085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.253:43791 every 8 connection(s)
I20260929 11:38:45.547412 29019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/data/info.pb
I20260929 11:38:45.551419 29086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:45.556236 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 29019
I20260929 11:38:45.556419 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-1/wal/instance
I20260929 11:38:45.556879 6183 external_mini_cluster.cc:1428] Setting key 0a3150836a717b5bae6a6ce9dfcf203e
I20260929 11:38:45.558135 29086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:45.578637 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.560646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.6.9.254:33315)
0929 11:38:45.560845 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:45.560850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:45.562201 (+ 1351us) server_negotiation.cc:407] Connection header received
0929 11:38:45.564018 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:45.564025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:45.564102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:45.564189 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:45.566068 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.566740 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.568726 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.569133 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.569224 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.573004 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:45.573009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:45.573012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:45.573046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:45.575496 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.576361 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.576367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.576372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.576513 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.576950 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.576955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.576959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.577257 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:45.577392 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:45.578094 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 11:38:45.578337 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:38:45.579542 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.561271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33315 (local address 127.0.0.1:55302)
0929 11:38:45.562033 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.562071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:45.563741 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:45.564419 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:45.564426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:45.565119 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:45.565876 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.565904 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.566889 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.566894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.568576 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.568590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.569333 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.569339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.569590 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.570255 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:45.570282 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:45.572773 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:45.575626 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.575635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.575650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.576199 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.576621 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.576624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.576627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.576833 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.577378 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:45.577387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:45.577708 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 11:38:45.577955 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 11:38:45.578447 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.6.9.252@KRBTEST.COM; defaulting to no policy
I20260929 11:38:45.581462 29086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.9.252@KRBTEST.COM" created.
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:45.594933 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.582201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47484)
0929 11:38:45.582398 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.582416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:45.582506 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:45.583408 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:45.583412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:45.583791 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:45.583908 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.583916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.585283 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.585289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.586086 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.586097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.586836 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.586842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.587013 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.587526 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:45.587542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:45.589465 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:45.592825 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.592829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.592832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.593173 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.593524 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.593527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.593529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.593647 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.594265 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:45.594269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:45.594379 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 11:38:45.594481 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:38:45.594662 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":2}
I20260929 11:38:45.595259 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.582439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.6.9.253:43791)
0929 11:38:45.583031 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:45.583037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:45.583061 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:38:45.583099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:45.583104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:45.583269 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:45.583391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:45.584062 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.585143 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.586211 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.586708 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.586824 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.589656 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:45.589665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:45.589676 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:45.589708 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:45.592706 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.593295 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.593303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.593306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.593439 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.593738 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.593742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.593745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.594146 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:45.594337 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:45.594652 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 11:38:45.594890 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":331,"threads_started":1}
I20260929 11:38:45.597559 29086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:45.598672 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.598260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58274)
0929 11:38:45.598442 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.598546 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:38:45.599016 29037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.600436 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.600142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58282)
0929 11:38:45.600269 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.600343 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:38:45.600634 29086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
Entry for principal kudu/127.6.9.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab.
I20260929 11:38:45.612486 6183 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
/tmp/dist-test-taskvWjXHB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:34781
--webserver_interface=127.6.9.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.9.212:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:38:45.649946 29086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 1
I20260929 11:38:45.651052 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.650601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58292)
0929 11:38:45.650803 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.650896 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:38:45.652088 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.651758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58302)
0929 11:38:45.651906 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.651977 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:38:45.652309 29086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.720578 29086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 2
I20260929 11:38:45.721832 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.721367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58308)
0929 11:38:45.721564 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.721680 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 11:38:45.722856 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.722512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58314)
0929 11:38:45.722669 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.722739 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 11:38:45.723066 29086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.743696 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } attempt: 5
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:45.758878 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.744387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47494)
0929 11:38:45.744685 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.744714 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:45.744865 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:45.745229 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:45.745235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:45.745785 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:45.746056 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.746074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.747085 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.747092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.748525 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:45.748543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.749332 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.749339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:45.749538 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.750194 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:45.750222 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:45.752776 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:45.756698 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.756703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.756706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.757056 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.757463 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:45.757466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:45.757468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:45.757588 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:45.758212 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:45.758215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:45.758337 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 11:38:45.758433 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:38:45.758627 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:38:45.759214 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.744634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.6.9.253:43791)
0929 11:38:45.744819 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:45.744823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:45.744842 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:38:45.744988 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:45.744992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:45.745071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:45.745152 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:45.746235 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.746926 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.748730 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:45.749152 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:45.749265 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:45.752989 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:45.752998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:45.753004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:45.753055 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:45.756510 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.757175 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.757183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.757187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.757354 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:45.757694 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:45.757699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:45.757703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:45.758108 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:45.758271 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:45.758596 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 11:38:45.758845 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:38:45.760170 29012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:38:45.761456 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.761025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58328)
0929 11:38:45.761196 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.761294 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:38:45.762398 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.762069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58334)
0929 11:38:45.762214 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.762297 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 11:38:45.762580 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.813787 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 1
I20260929 11:38:45.814271 29086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 3
I20260929 11:38:45.815099 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.814440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58346)
0929 11:38:45.814716 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.814864 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 11:38:45.815178 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.814779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58360)
0929 11:38:45.814948 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.815027 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 11:38:45.816293 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.815861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58370)
0929 11:38:45.816039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.816142 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:38:45.816416 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.815981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58372)
0929 11:38:45.816204 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.816286 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 11:38:45.816636 29086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
W20260929 11:38:45.816800 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.885032 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 2
I20260929 11:38:45.886253 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.885779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58380)
0929 11:38:45.885988 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.886093 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 11:38:45.887176 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.886849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58386)
0929 11:38:45.886987 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.887056 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 11:38:45.887377 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
W20260929 11:38:45.946565 29093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:38:45.946956 29093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:38:45.947078 29093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:38:45.947211 29093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:38:45.954784 29093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:38:45.954923 29093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:38:45.955016 29093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:38:45.955109 29093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:38:45.955207 29093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:38:45.959882 29086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 4
I20260929 11:38:45.961017 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.960498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58390)
0929 11:38:45.960751 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.960855 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:38:45.962013 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.961710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58402)
0929 11:38:45.961850 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.961912 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 11:38:45.962227 29086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:45.965902 29093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.9.212:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.9.254:33315,127.6.9.253:43791,127.6.9.252:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/krb5kdc/kudu_127.6.9.252.keytab
--principal=kudu/127.6.9.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.9.252:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.9.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:38:45.967252 29093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:38:45.968600 29093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:38:45.980460 29098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:45.980535 29099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:38:45.980741 29101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:38:45.981351 29093 server_base.cc:1034] running on GCE node
I20260929 11:38:45.982180 29093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:38:45.983808 29093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:38:45.985039 29093 hybrid_clock.cc:648] HybridClock initialized: now 1790681925984963 us; error 51 us; skew 500 ppm
I20260929 11:38:45.985406 29093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:38:45 dist-test-slave-c0l9 krb5kdc[28932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:38:45.988564 29093 init.cc:377] Logged in from keytab as kudu/127.6.9.252@KRBTEST.COM (short username kudu)
I20260929 11:38:45.990018 29093 webserver.cc:460] Webserver started at http://127.6.9.252:34717/ using document root <none> and password file <none>
I20260929 11:38:45.990707 29093 fs_manager.cc:359] Metadata directory not provided
I20260929 11:38:45.990792 29093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:38:45.991114 29093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:38:45.993562 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 3
I20260929 11:38:45.994175 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data/instance:
uuid: "901e419b33bd4e8682112337156977b2"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "6df79d3302cf53c4be97dc7f9645e04a"
server_key_iv: "551b57164a0b8df4d9d64092af2f56e3"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.994740 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.994158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58416)
0929 11:38:45.994439 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.994549 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 11:38:45.995060 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal/instance:
uuid: "901e419b33bd4e8682112337156977b2"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "6df79d3302cf53c4be97dc7f9645e04a"
server_key_iv: "551b57164a0b8df4d9d64092af2f56e3"
server_key_version: "encryptionkey@0"
I20260929 11:38:45.995993 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:45.995500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58418)
0929 11:38:45.995718 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:45.995801 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 11:38:45.996239 29012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.9.252:34781: Network error: Client connection negotiation failed: client connection to 127.6.9.252:34781: connect: Connection refused (error 111)
I20260929 11:38:46.000419 29093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:38:46.004022 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:38:46.005553 29093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:38:46.005724 29093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data,/tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal
uuid: "901e419b33bd4e8682112337156977b2"
format_stamp: "Formatted at 2026-09-29 11:38:45 on dist-test-slave-c0l9"
server_key: "6df79d3302cf53c4be97dc7f9645e04a"
server_key_iv: "551b57164a0b8df4d9d64092af2f56e3"
server_key_version: "encryptionkey@0"
I20260929 11:38:46.005928 29093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:38:46.027384 29093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:38:46.031913 29093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:38:46.032194 29093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:38:46.050206 29093 rpc_server.cc:307] RPC server started. Bound to: 127.6.9.252:34781
I20260929 11:38:46.050225 29159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.9.252:34781 every 8 connection(s)
I20260929 11:38:46.051501 29093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/data/info.pb
I20260929 11:38:46.055032 29160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:38:46.056577 6183 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu as pid 29093
I20260929 11:38:46.056783 6183 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvWjXHB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681566518919-6183-0/minicluster-data/master-2/wal/instance
I20260929 11:38:46.057221 6183 external_mini_cluster.cc:1428] Setting key 47ddb71928e579ee94bdf655bc6fca60
I20260929 11:38:46.063519 29160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:46.078493 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.058970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.6.9.252:34781)
0929 11:38:46.059729 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.059740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.059781 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:38:46.061380 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.061402 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.061804 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.062140 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.063522 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.064887 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.066260 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.067058 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.067195 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.071263 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.071272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.071283 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.071321 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.074402 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.075049 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.075056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.075062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.075202 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.075561 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.075565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.075567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.076054 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.076323 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.076790 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.077204 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260929 11:38:46.079744 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.065887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.6.9.254:33315)
0929 11:38:46.066102 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.066108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.066530 (+ 422us) server_negotiation.cc:407] Connection header received
0929 11:38:46.066700 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.066705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.066769 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.066849 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.068255 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.068989 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.070521 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.070963 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.071071 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.074519 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.074524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.074527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.074561 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.077070 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.077816 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.077820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.077822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.077949 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.078319 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.078322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.078324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.078568 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.078721 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.079192 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.079400 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:38:46.079793 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.065778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.254:33315 (local address 127.0.0.1:55304)
0929 11:38:46.066401 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:46.066442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:46.066587 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:46.066930 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:46.066937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:46.067777 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:46.068068 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.068080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.069139 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.069144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.070394 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.070409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.071172 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.071180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.071346 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.071954 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:46.071970 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:46.074338 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:46.077204 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.077213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.077219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.077713 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.078046 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.078051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.078054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.078228 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.078749 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:46.078758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:46.078905 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 11:38:46.079056 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 11:38:46.079359 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
I20260929 11:38:46.081522 29160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.254@KRBTEST.COM
I20260929 11:38:46.093871 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.082188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.253:43791 (local address 127.0.0.1:47496)
0929 11:38:46.082337 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:46.082353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:46.082434 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:46.082703 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:46.082706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:46.083003 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:46.083133 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.083140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.084085 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.084090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.085051 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.085062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.085722 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.085727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.085900 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.086470 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:46.086486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:46.088493 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:46.091628 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.091633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.091637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.091969 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.092313 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.092318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.092322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.092469 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.093000 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:46.093005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:46.093150 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 11:38:46.093265 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:38:46.093524 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 11:38:46.093931 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.082222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.6.9.253:43791)
0929 11:38:46.082410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.082416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.082436 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:38:46.082517 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.082522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.082596 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.082703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.083290 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.083949 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.085187 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.085595 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.085711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.088719 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.088727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.088733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.088772 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.091502 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.092075 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.092079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.092081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.092212 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.092564 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.092568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.092570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.092886 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.093023 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.093382 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.093583 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:38:46.094645 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.081454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.6.9.254:33315)
0929 11:38:46.081603 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.081608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.081625 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:38:46.081785 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.081791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.081855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.081930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.083254 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.083951 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.085155 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.085568 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.085672 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.089667 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.089673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.089676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.089718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.092132 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.092725 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.092730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.092734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.092863 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.093219 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.093224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.093227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.093469 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.093711 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.094128 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.094353 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:38:46.095302 29160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.252@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.9.253@KRBTEST.COM
I20260929 11:38:46.107434 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.096037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58440)
0929 11:38:46.096234 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:46.096250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:46.096347 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:46.096593 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:46.096597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:46.096996 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:46.097116 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.097123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.098005 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.098011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.098920 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.098931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.099554 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.099559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.099750 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.100319 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:46.100337 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:46.102374 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:46.105249 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.105255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.105259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.105581 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.105887 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.105892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.105896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.106016 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.106476 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:46.106481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:46.106612 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 11:38:46.106736 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 11:38:46.107005 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 11:38:46.107457 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.096018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.6.9.252:34781)
0929 11:38:46.096207 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.096213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.096306 (+ 93us) server_negotiation.cc:407] Connection header received
0929 11:38:46.096433 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.096439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.096503 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.096579 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.097235 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.097896 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.099036 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.099430 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.099533 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.102543 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.102548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.102551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.102579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.105137 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.105685 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.105690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.105694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.105795 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.106104 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.106110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.106113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.106375 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.106497 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.106824 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.107164 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":61}
I20260929 11:38:46.109977 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.096174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.6.9.253:43791)
0929 11:38:46.096326 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.096330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.096499 (+ 169us) server_negotiation.cc:407] Connection header received
0929 11:38:46.096640 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.096646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.096741 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.096821 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.098049 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.098891 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.100052 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.100466 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.100572 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.104705 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.104712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.104717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.104752 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.107125 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.108021 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.108027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.108031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.108233 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.108610 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.108615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.108618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.108959 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.109174 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.109510 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.109693 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:38:46.113070 29160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: Bootstrap starting.
I20260929 11:38:46.114545 6183 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:38:46.116287 29160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:46.117370 29160 log.cc:824] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:46.116873 28987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0a48099800b548e18ce6c26573da0e58"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55312
I20260929 11:38:46.120110 29160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: No bootstrap required, opened a new log
I20260929 11:38:46.127060 29160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.127388 29160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901e419b33bd4e8682112337156977b2, State: Initialized, Role: FOLLOWER
I20260929 11:38:46.127808 29160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.128613 29166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.128999 29166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.129731 29160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:38:46.139328 29177 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:38:46.139482 29177 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:38:46.136945 29093 master_runner.cc:186] Error getting master registration for 127.6.9.254:33315: OK, instance_id { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" instance_seqno: 1790681925037654 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:38:46.148003 29093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:38:46.173448 29012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 4
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681924, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.254@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:38:46.190559 29016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.174314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58452)
0929 11:38:46.174832 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:46.174855 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:46.175003 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:46.175419 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:46.175425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:46.175943 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:46.176162 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.176175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.177703 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.177708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.179147 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.179165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.180268 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.180275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.180475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.181194 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:46.181214 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:46.183473 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:46.187479 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.187485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.187489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.188067 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.188492 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.188497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.188499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.188747 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.189464 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:46.189469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:46.189650 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 11:38:46.189808 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 11:38:46.190188 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260929 11:38:46.190550 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.174407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.6.9.252:34781)
0929 11:38:46.174747 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.174755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.174954 (+ 199us) server_negotiation.cc:407] Connection header received
0929 11:38:46.175146 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.175151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.175241 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.175353 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.176339 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.177551 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.179318 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.180073 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.180240 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.183664 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.183671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.183676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.183718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.187319 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.188223 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.188228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.188230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.188370 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.188866 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.188872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.188876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.189343 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.189497 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.189937 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.190169 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 11:38:46.196532 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: Bootstrap starting.
I20260929 11:38:46.200140 29012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:46.201273 29012 log.cc:824] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: Log is configured to *not* fsync() on all Append() calls
I20260929 11:38:46.203966 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: No bootstrap required, opened a new log
I20260929 11:38:46.211428 29012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.211867 29012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a48099800b548e18ce6c26573da0e58, State: Initialized, Role: FOLLOWER
I20260929 11:38:46.212320 29012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.213322 29180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.213677 29180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.214423 29012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:46.214576 28987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:38:46.214792 28987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:46.218376 28987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.220937 29135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a48099800b548e18ce6c26573da0e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "901e419b33bd4e8682112337156977b2"
I20260929 11:38:46.221495 29135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:46.222334 28987 leader_election.cc:288] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [CANDIDATE]: Term 1 election: Requested vote from peers a35d3d1a82f4431c99613e390ee3b19d (127.6.9.253:43791), 901e419b33bd4e8682112337156977b2 (127.6.9.252:34781)
I20260929 11:38:46.225066 29135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a48099800b548e18ce6c26573da0e58 in term 1.
I20260929 11:38:46.225737 28963 leader_election.cc:302] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a48099800b548e18ce6c26573da0e58, 901e419b33bd4e8682112337156977b2; no voters:
I20260929 11:38:46.226227 29180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:38:46.237299 29180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [term 1 LEADER]: Becoming Leader. State: Replica: 0a48099800b548e18ce6c26573da0e58, State: Running, Role: LEADER
I20260929 11:38:46.237965 29180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.245656 29188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a48099800b548e18ce6c26573da0e58. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.246107 29188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:46.248595 29192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:38:46.259433 29192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:38:46.261379 29086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } attempt: 5
Sep 29 11:38:46 dist-test-slave-c0l9 krb5kdc[28932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681925, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.9.253@KRBTEST.COM for kudu/127.6.9.252@KRBTEST.COM
I20260929 11:38:46.279016 29088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.262063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.9.252:34781 (local address 127.0.0.1:58468)
0929 11:38:46.262456 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 11:38:46.262476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:38:46.262600 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:38:46.263566 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:38:46.263571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:38:46.264103 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:38:46.264318 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.264331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.265739 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.265745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.267167 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:38:46.267183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.268409 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.268416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:38:46.268637 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.269373 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:38:46.269397 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:38:46.272205 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:38:46.276106 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.276112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.276117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.276682 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.277126 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:38:46.277132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:38:46.277135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:38:46.277338 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:38:46.278004 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:38:46.278010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:38:46.278187 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 11:38:46.278338 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 11:38:46.278629 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260929 11:38:46.279999 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.262911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.6.9.252:34781)
0929 11:38:46.263092 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.263098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.263120 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:38:46.263195 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.263201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.263278 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.263370 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.264501 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.265565 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.267360 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.268075 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.268200 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.272420 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.272426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.272431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.272470 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.275946 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.276823 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.276828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.276831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.276997 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.277463 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.277468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.277472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.277805 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.277943 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.279401 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.279644 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:38:46.284310 29114 tablet.cc:2369] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:38:46.285894 29086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: Bootstrap starting.
I20260929 11:38:46.290624 29086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: Neither blocks nor log segments found. Creating new log.
I20260929 11:38:46.292009 29086 log.cc:824] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: Log is configured to *not* fsync() on all Append() calls
W20260929 11:38:46.292285 28966 tablet.cc:2369] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:38:46.297173 29086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: No bootstrap required, opened a new log
I20260929 11:38:46.308547 29086 raft_consensus.cc:348] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.309386 29086 raft_consensus.cc:729] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d3d1a82f4431c99613e390ee3b19d, State: Initialized, Role: FOLLOWER
I20260929 11:38:46.310096 29086 consensus_queue.cc:260] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } }
I20260929 11:38:46.337829 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.282607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.6.9.252:34781)
0929 11:38:46.290197 (+ 7590us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.290204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.315908 (+ 25704us) server_negotiation.cc:407] Connection header received
0929 11:38:46.316329 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.316336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.316424 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.316508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.319725 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.320873 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.323660 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.324379 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.324491 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.330007 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.330014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.330019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.330061 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.333501 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.334723 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.334729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.334732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.334906 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.335563 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.335568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.335571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.335976 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.336223 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.337143 (+ 920us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.337391 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6462,"thread_start_us":29324,"threads_started":1}
I20260929 11:38:46.350944 29090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.274762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.6.9.253:43791)
0929 11:38:46.274957 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.274963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.312963 (+ 38000us) server_negotiation.cc:407] Connection header received
0929 11:38:46.313395 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.313402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.313493 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.313567 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.319848 (+ 6281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.320928 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.325769 (+ 4841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.326709 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.326813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.338691 (+ 11878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.338700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.338704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.338752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.343893 (+ 5141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.346928 (+ 3035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.346935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.346939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.347111 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.347779 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.347817 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.347820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.348601 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.349036 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.349992 (+ 956us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.350371 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:38:46.351630 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:46.310991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.6.9.254:33315)
0929 11:38:46.311227 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.311233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.315770 (+ 4537us) server_negotiation.cc:407] Connection header received
0929 11:38:46.317325 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.317330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.317421 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.317494 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.322875 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.323968 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.327752 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:46.328465 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:46.328571 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:46.341610 (+ 13039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:38:46.341618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:38:46.341623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:38:46.341674 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:38:46.346743 (+ 5069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.347904 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.347909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.347912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.348210 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:38:46.348773 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:38:46.348778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:38:46.348781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:38:46.349166 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:38:46.349401 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:46.350824 (+ 1423us) server_negotiation.cc:299] Negotiation successful
0929 11:38:46.351064 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 11:38:46.353062 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.353489 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.355130 29135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [term 1 FOLLOWER]: Refusing update from remote peer 0a48099800b548e18ce6c26573da0e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:46.355381 29086 sys_catalog.cc:564] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:38:46.355477 29206 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:38:46.355747 29060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:38:46.356261 29188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:46.356172 29061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a48099800b548e18ce6c26573da0e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a35d3d1a82f4431c99613e390ee3b19d"
I20260929 11:38:46.360059 29060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [term 1 FOLLOWER]: Refusing update from remote peer 0a48099800b548e18ce6c26573da0e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:38:46.361147 29188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:38:46.381448 29166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a48099800b548e18ce6c26573da0e58. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.381834 29166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.372359 28945 master_runner.cc:186] Error getting master registration for 127.6.9.253:43791: OK, instance_id { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" instance_seqno: 1790681925541132 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:38:46.383284 28945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:38:46.386929 29205 mvcc.cc:205] Tried to move back new op lower bound from 7334633170269270016 to 7334633169876795392. Current Snapshot: MvccSnapshot[applied={T|T < 7334633170269270016}]
I20260929 11:38:46.392112 29216 mvcc.cc:205] Tried to move back new op lower bound from 7334633170269270016 to 7334633169876795392. Current Snapshot: MvccSnapshot[applied={T|T < 7334633170269270016}]
I20260929 11:38:46.400269 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a48099800b548e18ce6c26573da0e58. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.400626 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.401068 29166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.401324 29166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.406070 29061 raft_consensus.cc:2396] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a48099800b548e18ce6c26573da0e58 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 11:38:46.411612 29192 catalog_manager.cc:1347] Generated new cluster ID: c0d6343c187b459ca592de55132802c8
I20260929 11:38:46.411731 29192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:38:46.413237 29188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.413482 29188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:46.413736 29180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.413965 29180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58 [sys.catalog]: This master's current role is: LEADER
I20260929 11:38:46.417737 29203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a48099800b548e18ce6c26573da0e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a48099800b548e18ce6c26573da0e58" member_type: VOTER last_known_addr { host: "127.6.9.254" port: 33315 } } peers { permanent_uuid: "a35d3d1a82f4431c99613e390ee3b19d" member_type: VOTER last_known_addr { host: "127.6.9.253" port: 43791 } } peers { permanent_uuid: "901e419b33bd4e8682112337156977b2" member_type: VOTER last_known_addr { host: "127.6.9.252" port: 34781 } } }
I20260929 11:38:46.417958 29203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:38:46.433276 29192 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:38:46.435614 29192 catalog_manager.cc:1504] Loading token signing keys...
W20260929 11:38:46.444738 29223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:38:46.444903 29223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:38:46.467962 29192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a48099800b548e18ce6c26573da0e58: Generated new TSK 0
I20260929 11:38:46.469184 29192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:38:46.616950 29163 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.605054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.6.9.252:34781)
0929 11:38:46.605312 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.605319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.613700 (+ 8381us) server_negotiation.cc:407] Connection header received
0929 11:38:46.614322 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.614329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.614432 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.614511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.616655 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58492: BlockingRecv error: recv got EOF from 127.0.0.1:58492 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 11:38:46.619693 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.598666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.6.9.253:43791)
0929 11:38:46.598912 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.598919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.609297 (+ 10378us) server_negotiation.cc:407] Connection header received
0929 11:38:46.618048 (+ 8751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.618055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.618177 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.618258 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.619444 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47516: BlockingRecv error: recv got EOF from 127.0.0.1:47516 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 11:38:46.620146 29015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.610266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.6.9.254:33315)
0929 11:38:46.610477 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.610483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.615861 (+ 5378us) server_negotiation.cc:407] Connection header received
0929 11:38:46.616015 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.616021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.616105 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.616227 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.619892 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55330: BlockingRecv error: recv got EOF from 127.0.0.1:55330 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 11:38:46.663064 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.649747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.6.9.253:43791)
0929 11:38:46.649943 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.649949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.661029 (+ 11080us) server_negotiation.cc:407] Connection header received
0929 11:38:46.661409 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.661416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.661567 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.661653 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.662773 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47532: BlockingRecv error: recv got EOF from 127.0.0.1:47532 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:38:46.704892 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.698534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.6.9.253:43791)
0929 11:38:46.698736 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.698742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.703330 (+ 4588us) server_negotiation.cc:407] Connection header received
0929 11:38:46.703528 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.703533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.703635 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.703780 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.704623 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47546: BlockingRecv error: recv got EOF from 127.0.0.1:47546 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:38:46.716912 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.714434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.6.9.253:43791)
0929 11:38:46.714615 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.714622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.715256 (+ 634us) server_negotiation.cc:407] Connection header received
0929 11:38:46.715453 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.715460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.715574 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.715695 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.716614 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47550: BlockingRecv error: recv got EOF from 127.0.0.1:47550 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:38:46.726737 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.724886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.6.9.253:43791)
0929 11:38:46.725117 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.725123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.725203 (+ 80us) server_negotiation.cc:407] Connection header received
0929 11:38:46.725494 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.725500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.725597 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.725708 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.726493 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47564: BlockingRecv error: recv got EOF from 127.0.0.1:47564 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260929 11:38:46.744745 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.742525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.6.9.253:43791)
0929 11:38:46.742733 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.742739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.743253 (+ 514us) server_negotiation.cc:407] Connection header received
0929 11:38:46.743471 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.743477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.743577 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.743701 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.744466 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47570: BlockingRecv error: recv got EOF from 127.0.0.1:47570 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:38:46.767278 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.764787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.6.9.253:43791)
0929 11:38:46.765003 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.765009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.765540 (+ 531us) server_negotiation.cc:407] Connection header received
0929 11:38:46.765766 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.765772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.765876 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.765996 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.766992 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47580: BlockingRecv error: recv got EOF from 127.0.0.1:47580 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:38:46.813171 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.810486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.6.9.253:43791)
0929 11:38:46.810695 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.810701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.811310 (+ 609us) server_negotiation.cc:407] Connection header received
0929 11:38:46.811596 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.811602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.811725 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.811835 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.812889 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47586: BlockingRecv error: recv got EOF from 127.0.0.1:47586 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 11:38:46.883941 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:46.881728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.6.9.253:43791)
0929 11:38:46.881956 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:46.881964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:46.882421 (+ 457us) server_negotiation.cc:407] Connection header received
0929 11:38:46.882651 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:46.882656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:46.882731 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:46.882809 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:46.883714 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47594: BlockingRecv error: recv got EOF from 127.0.0.1:47594 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 11:38:47.017256 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:47.015334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.6.9.253:43791)
0929 11:38:47.015565 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.015571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.015873 (+ 302us) server_negotiation.cc:407] Connection header received
0929 11:38:47.016068 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.016072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.016165 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.016252 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:47.017005 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47598: BlockingRecv error: recv got EOF from 127.0.0.1:47598 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:38:47.141531 29177 catalog_manager.cc:1259] Loaded cluster ID: c0d6343c187b459ca592de55132802c8
I20260929 11:38:47.141645 29177 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: loading cluster ID for follower catalog manager: success
I20260929 11:38:47.144132 29177 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: acquiring CA information for follower catalog manager: success
I20260929 11:38:47.145956 29177 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 901e419b33bd4e8682112337156977b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:38:47.280068 29090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:38:47.278427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.6.9.253:43791)
0929 11:38:47.278653 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.278659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.278795 (+ 136us) server_negotiation.cc:407] Connection header received
0929 11:38:47.279005 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.279010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.279084 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.279170 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:38:47.279894 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47608: BlockingRecv error: recv got EOF from 127.0.0.1:47608 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:38:47.446655 29223 catalog_manager.cc:1259] Loaded cluster ID: c0d6343c187b459ca592de55132802c8
I20260929 11:38:47.446762 29223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: loading cluster ID for follower catalog manager: success
I20260929 11:38:47.449285 29223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: acquiring CA information for follower catalog manager: success
I20260929 11:38:47.450989 29223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a35d3d1a82f4431c99613e390ee3b19d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:38:47.802919 29290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:47.797258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47610 (local address 127.6.9.253:43791)
0929 11:38:47.797750 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.797756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.797777 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:38:47.797937 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.797941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.798101 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.798220 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:47.798674 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.799378 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.800716 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.801138 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.801260 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:47.801751 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:47.802294 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:47.802431 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:47.802540 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 11:38:47.802637 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260929 11:38:47.814957 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:47.808565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.6.9.252:34781)
0929 11:38:47.809074 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.809078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.809101 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:38:47.809219 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.809223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.809391 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.809570 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:47.810035 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.810792 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.812273 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.812768 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.812880 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:47.813482 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:47.814075 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:47.814205 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:47.814395 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 11:38:47.814480 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260929 11:38:47.829118 29290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:47.821872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.6.9.253:43791)
0929 11:38:47.822079 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.822086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.822703 (+ 617us) server_negotiation.cc:407] Connection header received
0929 11:38:47.823043 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.823050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.823161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.823238 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:47.824114 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.825140 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.827054 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.827458 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.827577 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:47.828292 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:47.828438 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:47.828490 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:47.828739 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 11:38:47.828826 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:38:47.830322 29296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:47.821945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.6.9.252:34781)
0929 11:38:47.823034 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.823040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.823594 (+ 554us) server_negotiation.cc:407] Connection header received
0929 11:38:47.823778 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.823784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.823929 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.824648 (+ 719us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:47.824861 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.825813 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.827544 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.828243 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.828358 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:47.828844 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:47.829148 (+ 304us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:47.829230 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:47.829374 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 11:38:47.829456 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955}
I20260929 11:38:47.830776 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:38:47.821579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.6.9.254:33315)
0929 11:38:47.822851 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0929 11:38:47.822859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:38:47.824371 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 11:38:47.824583 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:38:47.824588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:38:47.824868 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:38:47.825038 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:38:47.825615 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.826666 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.828312 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:38:47.828970 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:38:47.829085 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:38:47.829701 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:38:47.830133 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:38:47.830196 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 11:38:47.830414 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 11:38:47.830509 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":983,"threads_started":1}
I20260929 11:38:47.835705 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 28945
I20260929 11:38:47.836691 29007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:47.947429 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 28945
W20260929 11:38:47.965173 29035 connection.cc:535] server connection from 127.0.0.1:47494 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:38:47.965533 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 29019
I20260929 11:38:47.966673 29081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:48.077481 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 29019
I20260929 11:38:48.094151 6183 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu pid 29093
I20260929 11:38:48.095250 29155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:38:48.201490 6183 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvWjXHB/build/asan/bin/kudu with pid 29093
2026-09-29T11:38:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4838 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9599 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361637 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST